[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:49 dist-test-slave-n5vv krb5kdc[2335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:49 dist-test-slave-n5vv krb5kdc[2335](info): set up 2 sockets
Jun 20 14:59:49 dist-test-slave-n5vv krb5kdc[2335](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 14:59:50.390113 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.007s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 14:59:50 dist-test-slave-n5vv krb5kdc[2335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 14:59:50.571455 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45655
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:45655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:50.898447 2348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:50.898795 2348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:50.898970 2348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:50.906154 2348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:50.906317 2348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:50.906423 2348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:50.906524 2348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:50.906610 2348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:50.916700 2348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:45655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:50.918113 2348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:50.919471 2348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:50.932215 2353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:50.932189 2354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:50.932286 2356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:50.932862 2348 server_base.cc:1034] running on GCE node
I20260620 14:59:50.933790 2348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:50.935117 2348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:50.936311 2348 hybrid_clock.cc:648] HybridClock initialized: now 1781967590936257 us; error 51 us; skew 500 ppm
I20260620 14:59:50.936679 2348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:50 dist-test-slave-n5vv krb5kdc[2335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967590, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:50.940171 2348 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 14:59:50.941617 2348 webserver.cc:460] Webserver started at http://127.2.70.62:41043/ using document root <none> and password file <none>
I20260620 14:59:50.942292 2348 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:50.942421 2348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:50.942734 2348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:50.945817 2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "d90e7278ced9412588355edc23b6de73"
format_stamp: "Formatted at 2026-06-20 14:59:50 on dist-test-slave-n5vv"
server_key: "53e61e7bd24fcec1fdda4677f199c75c"
server_key_iv: "a4e79921bb27e0d2d4a307804e3085bd"
server_key_version: "encryptionkey@0"
I20260620 14:59:50.946611 2348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "d90e7278ced9412588355edc23b6de73"
format_stamp: "Formatted at 2026-06-20 14:59:50 on dist-test-slave-n5vv"
server_key: "53e61e7bd24fcec1fdda4677f199c75c"
server_key_iv: "a4e79921bb27e0d2d4a307804e3085bd"
server_key_version: "encryptionkey@0"
I20260620 14:59:50.952211 2348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:59:50.955514 2363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:50.956797 2348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:59:50.956976 2348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "d90e7278ced9412588355edc23b6de73"
format_stamp: "Formatted at 2026-06-20 14:59:50 on dist-test-slave-n5vv"
server_key: "53e61e7bd24fcec1fdda4677f199c75c"
server_key_iv: "a4e79921bb27e0d2d4a307804e3085bd"
server_key_version: "encryptionkey@0"
I20260620 14:59:50.957175 2348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:50.985304 2348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:50.989732 2348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:50.990463 2348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:51.009671 2348 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:45655
I20260620 14:59:51.009655 2414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:45655 every 8 connection(s)
I20260620 14:59:51.011063 2348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:51.014878 2415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:51.018422 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 2348
I20260620 14:59:51.018735 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 14:59:51.021106 2328 external_mini_cluster.cc:1428] Setting key 79cc3451f865e4ebd7f06c5ddbb3ed76
I20260620 14:59:51.028138 2415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73: Bootstrap starting.
I20260620 14:59:51.031404 2415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:51.032335 2415 log.cc:824] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:51.034904 2415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73: No bootstrap required, opened a new log
I20260620 14:59:51.044818 2415 raft_consensus.cc:348] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90e7278ced9412588355edc23b6de73" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45655 } }
I20260620 14:59:51.045258 2415 raft_consensus.cc:374] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:51.045373 2415 raft_consensus.cc:729] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90e7278ced9412588355edc23b6de73, State: Initialized, Role: FOLLOWER
I20260620 14:59:51.045980 2415 consensus_queue.cc:260] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90e7278ced9412588355edc23b6de73" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45655 } }
I20260620 14:59:51.046200 2415 raft_consensus.cc:388] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:51.046350 2415 raft_consensus.cc:482] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:51.046520 2415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:51.048544 2415 raft_consensus.cc:504] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90e7278ced9412588355edc23b6de73" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45655 } }
I20260620 14:59:51.049011 2415 leader_election.cc:302] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d90e7278ced9412588355edc23b6de73; no voters:
Jun 20 14:59:51 dist-test-slave-n5vv krb5kdc[2335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 14:59:51.050096 2415 leader_election.cc:288] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:51.050232 2420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:51.051287 2420 raft_consensus.cc:686] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [term 1 LEADER]: Becoming Leader. State: Replica: d90e7278ced9412588355edc23b6de73, State: Running, Role: LEADER
I20260620 14:59:51.051877 2420 consensus_queue.cc:237] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90e7278ced9412588355edc23b6de73" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45655 } }
I20260620 14:59:51.052376 2415 sys_catalog.cc:564] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:51.054844 2421 sys_catalog.cc:455] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d90e7278ced9412588355edc23b6de73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90e7278ced9412588355edc23b6de73" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45655 } } }
I20260620 14:59:51.055208 2421 sys_catalog.cc:458] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:51.057983 2422 sys_catalog.cc:455] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d90e7278ced9412588355edc23b6de73. Latest consensus state: current_term: 1 leader_uuid: "d90e7278ced9412588355edc23b6de73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d90e7278ced9412588355edc23b6de73" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45655 } } }
I20260620 14:59:51.058344 2422 sys_catalog.cc:458] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:51.063602 2425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:51.068645 2425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:51.069842 2417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:51.025845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.2.70.62:45655)
0620 14:59:51.026788 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:51.026802 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:51.028146 (+ 1344us) server_negotiation.cc:407] Connection header received
0620 14:59:51.037184 (+ 9038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:51.037209 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:51.037714 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:51.038155 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:51.040611 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:51.042702 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:51.044282 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:51.045289 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:51.045476 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:51.051853 (+ 6377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:51.051889 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:51.051906 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:51.051947 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:51.061075 (+ 9128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:51.062658 (+ 1583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:51.062670 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:51.062681 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:51.062933 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:51.063648 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:51.063655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:51.063658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:51.064336 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:51.064655 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:51.067514 (+ 2859us) server_negotiation.cc:299] Negotiation successful
0620 14:59:51.068054 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":398,"threads_started":1}
I20260620 14:59:51.081347 2425 catalog_manager.cc:1347] Generated new cluster ID: 505efbc1d3bd4f038bbc1954fe9f9305
I20260620 14:59:51.081516 2425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:51.105198 2425 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:51.107254 2425 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:51.125643 2425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d90e7278ced9412588355edc23b6de73: Generated new TSK 0
I20260620 14:59:51.126554 2425 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 14:59:51.181367 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45655
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:51.544520 2441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:51.544816 2441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:51.545017 2441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:51.553388 2441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:51.553563 2441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:51.553776 2441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 14:59:51.563874 2441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:51.565243 2441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:51.566642 2441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:51.580055 2446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:51.580068 2449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:51.580034 2447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:51.580904 2441 server_base.cc:1034] running on GCE node
I20260620 14:59:51.581686 2441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:51.582638 2441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:51.583866 2441 hybrid_clock.cc:648] HybridClock initialized: now 1781967591583816 us; error 47 us; skew 500 ppm
I20260620 14:59:51.584188 2441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:51 dist-test-slave-n5vv krb5kdc[2335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:51.587402 2441 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 14:59:51.589036 2441 webserver.cc:460] Webserver started at http://127.2.70.1:45779/ using document root <none> and password file <none>
I20260620 14:59:51.589841 2441 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:51.589948 2441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:51.590221 2441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:51.593089 2441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "8687b93d823b4e84b668c063d99fa76a"
format_stamp: "Formatted at 2026-06-20 14:59:51 on dist-test-slave-n5vv"
server_key: "e0af565ce609485267118f886a3a9a85"
server_key_iv: "300d451c037e73cbff50ad0ad64695f5"
server_key_version: "encryptionkey@0"
I20260620 14:59:51.593900 2441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "8687b93d823b4e84b668c063d99fa76a"
format_stamp: "Formatted at 2026-06-20 14:59:51 on dist-test-slave-n5vv"
server_key: "e0af565ce609485267118f886a3a9a85"
server_key_iv: "300d451c037e73cbff50ad0ad64695f5"
server_key_version: "encryptionkey@0"
I20260620 14:59:51.599704 2441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260620 14:59:51.603210 2456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:51.604612 2441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:59:51.604779 2441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "8687b93d823b4e84b668c063d99fa76a"
format_stamp: "Formatted at 2026-06-20 14:59:51 on dist-test-slave-n5vv"
server_key: "e0af565ce609485267118f886a3a9a85"
server_key_iv: "300d451c037e73cbff50ad0ad64695f5"
server_key_version: "encryptionkey@0"
I20260620 14:59:51.604983 2441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:51.634963 2441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:51.639139 2441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:51.639436 2441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:51.640489 2441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:51.642338 2441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:51.642412 2441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:51.642534 2441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:51.642603 2441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:51.676445 2441 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:45479
I20260620 14:59:51.676486 2568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:45479 every 8 connection(s)
I20260620 14:59:51.678038 2441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 14:59:51.680997 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 2441
I20260620 14:59:51.681249 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 14:59:51.681952 2328 external_mini_cluster.cc:1428] Setting key ca857c76cc2362784d3ba5a24010b0af
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Jun 20 14:59:51 dist-test-slave-n5vv krb5kdc[2335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967591, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
I20260620 14:59:51.706458 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:51.681173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:42733 (local address 127.2.70.62:45655)
0620 14:59:51.689332 (+ 8159us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:51.689336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:51.689353 (+ 17us) server_negotiation.cc:407] Connection header received
0620 14:59:51.689405 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:51.689408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:51.689467 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:51.689622 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:51.691909 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:51.692598 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:51.695310 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:51.695712 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:51.695808 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:51.700485 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:51.700490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:51.700493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:51.700525 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:51.703125 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:51.704107 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:51.704111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:51.704113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:51.704219 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:51.704721 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:51.704724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:51.704726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:51.704979 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:51.705084 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:51.706024 (+ 940us) server_negotiation.cc:299] Negotiation successful
0620 14:59:51.706194 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8052,"thread_start_us":7951,"threads_started":1}
I20260620 14:59:51.707679 2572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:51.682016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45655 (local address 127.2.70.1:42733)
0620 14:59:51.683020 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:51.683073 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:51.685545 (+ 2472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:51.689947 (+ 4402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:51.689963 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:51.690831 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:51.691714 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:51.691742 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:51.692765 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:51.692773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:51.695155 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:51.695174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:51.695972 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:51.695983 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:51.696347 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:51.697122 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:51.697159 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:51.700269 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:51.703269 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:51.703281 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:51.703303 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:51.703962 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:51.704328 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:51.704336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:51.704342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:51.704607 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:51.705102 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:51.705113 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:51.705619 (+ 506us) client_negotiation.cc:769] Sending connection context
0620 14:59:51.705921 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 14:59:51.706541 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":399,"threads_started":1}
I20260620 14:59:51.709228 2569 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45655
I20260620 14:59:51.709625 2569 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:51.710420 2569 heartbeater.cc:511] Master 127.2.70.62:45655 requested a full tablet report, sending...
I20260620 14:59:51.712540 2380 ts_manager.cc:194] Registered new tserver with Master: 8687b93d823b4e84b668c063d99fa76a (127.2.70.1:45479)
I20260620 14:59:51.714039 2380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:42733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 14:59:51.724855 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45655
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:52.058265 2575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:52.058619 2575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:52.058791 2575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:52.066331 2575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:52.066491 2575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:52.066671 2575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 14:59:52.076838 2575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:52.078598 2575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:52.080099 2575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:52.094049 2583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:52.094025 2580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:52.094023 2581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:52.094597 2575 server_base.cc:1034] running on GCE node
I20260620 14:59:52.095232 2575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:52.096158 2575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:52.097405 2575 hybrid_clock.cc:648] HybridClock initialized: now 1781967592097367 us; error 43 us; skew 500 ppm
I20260620 14:59:52.097766 2575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:52.100853 2575 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 14:59:52.102509 2575 webserver.cc:460] Webserver started at http://127.2.70.2:36829/ using document root <none> and password file <none>
I20260620 14:59:52.103224 2575 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:52.103371 2575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:52.103690 2575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:52.106546 2575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "f9eae6be0d264c868e99db46743a515d"
format_stamp: "Formatted at 2026-06-20 14:59:52 on dist-test-slave-n5vv"
server_key: "5a532372ba892588c7abc1c00a1a15ae"
server_key_iv: "9eab9d57ad192285709eaa5b78f17265"
server_key_version: "encryptionkey@0"
I20260620 14:59:52.107303 2575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "f9eae6be0d264c868e99db46743a515d"
format_stamp: "Formatted at 2026-06-20 14:59:52 on dist-test-slave-n5vv"
server_key: "5a532372ba892588c7abc1c00a1a15ae"
server_key_iv: "9eab9d57ad192285709eaa5b78f17265"
server_key_version: "encryptionkey@0"
I20260620 14:59:52.113163 2575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 14:59:52.116688 2590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:52.118122 2575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 14:59:52.118306 2575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "f9eae6be0d264c868e99db46743a515d"
format_stamp: "Formatted at 2026-06-20 14:59:52 on dist-test-slave-n5vv"
server_key: "5a532372ba892588c7abc1c00a1a15ae"
server_key_iv: "9eab9d57ad192285709eaa5b78f17265"
server_key_version: "encryptionkey@0"
I20260620 14:59:52.118546 2575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:52.152271 2575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:52.156919 2575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:52.157223 2575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:52.158257 2575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:52.159601 2575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:52.159708 2575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:52.159859 2575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:52.159932 2575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:52.195117 2575 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:45225
I20260620 14:59:52.195153 2702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:45225 every 8 connection(s)
I20260620 14:59:52.196446 2575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 14:59:52.202425 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 2575
I20260620 14:59:52.202615 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 14:59:52.203158 2328 external_mini_cluster.cc:1428] Setting key 7079095890a30fa2ed81ebea20303f84
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 14:59:52.220242 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:52.199228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:53239 (local address 127.2.70.62:45655)
0620 14:59:52.199410 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:52.199415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:52.200655 (+ 1240us) server_negotiation.cc:407] Connection header received
0620 14:59:52.203325 (+ 2670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:52.203330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:52.203401 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:52.203492 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:52.205366 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.206126 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.208737 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.209231 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.209330 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:52.213134 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:52.213143 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:52.213151 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:52.213196 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:52.216849 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.217678 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.217686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.217692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.217861 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.218302 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.218310 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.218316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.218695 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:52.218856 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:52.219550 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 14:59:52.219805 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 14:59:52.220983 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:52.199802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45655 (local address 127.2.70.2:53239)
0620 14:59:52.200476 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:52.200526 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:52.202995 (+ 2469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:52.203798 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:52.203807 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:52.204468 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:52.205191 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:52.205218 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.206278 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.206287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:52.208581 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:52.208598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.209467 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.209476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:52.209787 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:52.210489 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:52.210524 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:52.212883 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:52.217025 (+ 4142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:52.217033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:52.217048 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:52.217547 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:52.217989 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:52.217992 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:52.217995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:52.218174 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:52.218824 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:52.218830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:52.219161 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 14:59:52.219385 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 14:59:52.219778 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":276,"threads_started":1}
I20260620 14:59:52.222646 2703 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45655
I20260620 14:59:52.223037 2703 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
I20260620 14:59:52.223989 2703 heartbeater.cc:511] Master 127.2.70.62:45655 requested a full tablet report, sending...
I20260620 14:59:52.225661 2380 ts_manager.cc:194] Registered new tserver with Master: f9eae6be0d264c868e99db46743a515d (127.2.70.2:45225)
I20260620 14:59:52.226902 2380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:53239
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 14:59:52.245393 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45655
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:52.578015 2708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:52.578298 2708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:52.578470 2708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:52.586180 2708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:52.586292 2708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:52.586511 2708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 14:59:52.596385 2708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:52.597721 2708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:52.599069 2708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:52.612427 2713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:52.612392 2714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:52.612401 2716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:52.613684 2708 server_base.cc:1034] running on GCE node
I20260620 14:59:52.614249 2708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:52.615123 2708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:52.616317 2708 hybrid_clock.cc:648] HybridClock initialized: now 1781967592616274 us; error 39 us; skew 500 ppm
I20260620 14:59:52.616654 2708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:52.620149 2708 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 14:59:52.621682 2708 webserver.cc:460] Webserver started at http://127.2.70.3:38051/ using document root <none> and password file <none>
I20260620 14:59:52.622404 2708 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:52.622501 2708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:52.622816 2708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:52.625782 2708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "b898a805e79d464a8cfb84a3381cc3a1"
format_stamp: "Formatted at 2026-06-20 14:59:52 on dist-test-slave-n5vv"
server_key: "1e3842d5f7a0de0508c2ea5d05272a37"
server_key_iv: "c9f00d86b66868169243ea688efa03fa"
server_key_version: "encryptionkey@0"
I20260620 14:59:52.626507 2708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "b898a805e79d464a8cfb84a3381cc3a1"
format_stamp: "Formatted at 2026-06-20 14:59:52 on dist-test-slave-n5vv"
server_key: "1e3842d5f7a0de0508c2ea5d05272a37"
server_key_iv: "c9f00d86b66868169243ea688efa03fa"
server_key_version: "encryptionkey@0"
I20260620 14:59:52.631803 2708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:59:52.635424 2723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:52.636979 2708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:59:52.637187 2708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "b898a805e79d464a8cfb84a3381cc3a1"
format_stamp: "Formatted at 2026-06-20 14:59:52 on dist-test-slave-n5vv"
server_key: "1e3842d5f7a0de0508c2ea5d05272a37"
server_key_iv: "c9f00d86b66868169243ea688efa03fa"
server_key_version: "encryptionkey@0"
I20260620 14:59:52.637434 2708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:52.665625 2708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:52.670660 2708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:52.670987 2708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:52.672019 2708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:52.673408 2708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:52.673520 2708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:52.673672 2708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:52.673734 2708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:52.710201 2708 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:41793
I20260620 14:59:52.710242 2835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:41793 every 8 connection(s)
I20260620 14:59:52.711483 2708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 14:59:52.711803 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 2708
I20260620 14:59:52.712390 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 14:59:52.712926 2328 external_mini_cluster.cc:1428] Setting key 341268ffdd8af42f22e8c0772f0d001d
I20260620 14:59:52.716825 2569 heartbeater.cc:503] Master 127.2.70.62:45655 was elected leader, sending a full tablet report...
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 14:59:52.732226 2571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:52.714815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:49221 (local address 127.2.70.62:45655)
0620 14:59:52.714979 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:52.714984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:52.716426 (+ 1442us) server_negotiation.cc:407] Connection header received
0620 14:59:52.718494 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:52.718498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:52.718558 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:52.718629 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:52.720040 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.720799 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.722526 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.722893 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.723042 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:52.726358 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:52.726362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:52.726366 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:52.726395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:52.729375 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.730227 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.730230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.730233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.730351 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.730744 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.730747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.730749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.731004 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:52.731118 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:52.731822 (+ 704us) server_negotiation.cc:299] Negotiation successful
0620 14:59:52.731971 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 14:59:52.733345 2838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:52.715539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45655 (local address 127.2.70.3:49221)
0620 14:59:52.716278 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:52.716320 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:52.718206 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:52.718828 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:52.718836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:52.719327 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:52.719900 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:52.719917 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.720915 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.720919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:52.722413 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:52.722424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.723143 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.723154 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:52.723399 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:52.723995 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:52.724016 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:52.726165 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:52.729582 (+ 3417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:52.729592 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:52.729609 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:52.730109 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:52.730449 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:52.730453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:52.730455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:52.730643 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:52.731129 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:52.731135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:52.731474 (+ 339us) client_negotiation.cc:769] Sending connection context
0620 14:59:52.731721 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 14:59:52.732123 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":322,"threads_started":1}
I20260620 14:59:52.734670 2836 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45655
I20260620 14:59:52.735358 2836 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:52.736058 2836 heartbeater.cc:511] Master 127.2.70.62:45655 requested a full tablet report, sending...
I20260620 14:59:52.737382 2380 ts_manager.cc:194] Registered new tserver with Master: b898a805e79d464a8cfb84a3381cc3a1 (127.2.70.3:41793)
I20260620 14:59:52.738206 2380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:49221
I20260620 14:59:52.741838 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 14:59:52.785799 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:52.770326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.2.70.1:45479)
0620 14:59:52.770981 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:52.770988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:52.771067 (+ 79us) server_negotiation.cc:407] Connection header received
0620 14:59:52.771233 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:52.771239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:52.771449 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:52.771617 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:52.772983 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.774043 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.775243 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.775972 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.776068 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:52.780430 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:52.780441 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:52.780456 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:52.780488 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:52.783359 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.783938 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.783944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.783947 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.784064 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.784395 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.784398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.784399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.784733 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:52.784942 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:52.785167 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 14:59:52.785361 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":359,"threads_started":1}
W20260620 14:59:52.787061 2484 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53964
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 14:59:52.830116 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:52.816806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53980 (local address 127.2.70.1:45479)
0620 14:59:52.817031 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:52.817036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:52.817131 (+ 95us) server_negotiation.cc:407] Connection header received
0620 14:59:52.817279 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:52.817283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:52.817372 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:52.817458 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:52.818797 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.819481 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.820697 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.821093 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.821187 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:52.824865 (+ 3678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:52.824869 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:52.824872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:52.824906 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:52.827502 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.828075 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.828079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.828081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.828197 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.828494 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.828497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.828499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.829167 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:52.829332 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:52.829646 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 14:59:52.829799 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 14:59:52.831640 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 2441
I20260620 14:59:52.843641 2843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:52.832657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.2.70.1:45479)
0620 14:59:52.832850 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:52.832856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:52.832954 (+ 98us) server_negotiation.cc:407] Connection header received
0620 14:59:52.833080 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:52.833084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:52.833133 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:52.833203 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:52.834278 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.834937 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.836072 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.836464 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.836566 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:52.838633 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:52.838637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:52.838640 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:52.838666 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:52.841102 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.841790 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.841797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.841802 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.841933 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.842337 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.842342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.842347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.842600 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:52.842802 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:52.843065 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 14:59:52.843250 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 14:59:52.844367 2564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:52.927822 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 2441
I20260620 14:59:52.944442 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 2575
Jun 20 14:59:52 dist-test-slave-n5vv krb5kdc[2335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 14:59:52.960939 2850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:52.945429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.2.70.2:45225)
0620 14:59:52.946003 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:52.946009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:52.946035 (+ 26us) server_negotiation.cc:407] Connection header received
0620 14:59:52.946099 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:52.946105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:52.946309 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:52.946491 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:52.947669 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.948746 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.950050 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:52.950610 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:52.950753 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:52.955212 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:52.955224 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:52.955239 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:52.955274 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:52.958095 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.958782 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.958790 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.958792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.958918 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:52.959343 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:52.959347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:52.959349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:52.959696 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:52.959987 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:52.960277 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 14:59:52.960552 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":311,"threads_started":1}
I20260620 14:59:52.962302 2698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:53.045456 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 2575
I20260620 14:59:53.067137 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 2708
Jun 20 14:59:53 dist-test-slave-n5vv krb5kdc[2335](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967592, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 14:59:53.082618 2852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:53.068122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.2.70.3:41793)
0620 14:59:53.068753 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:53.068760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:53.068788 (+ 28us) server_negotiation.cc:407] Connection header received
0620 14:59:53.068847 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:53.068853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:53.069083 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:53.069216 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:53.070415 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:53.071422 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:53.072626 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:53.073153 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:53.073244 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:53.077236 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:53.077246 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:53.077259 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:53.077294 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:53.079927 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:53.080627 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:53.080634 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:53.080636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:53.080747 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:53.081046 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:53.081049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:53.081051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:53.081371 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:53.081772 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:53.082007 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 14:59:53.082197 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":332,"threads_started":1}
I20260620 14:59:53.083925 2831 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:53.166715 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 2708
I20260620 14:59:53.183089 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 2348
I20260620 14:59:53.184432 2410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:53.285665 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 2348
2026-06-20T14:59:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4005 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:53 dist-test-slave-n5vv krb5kdc[2857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:53 dist-test-slave-n5vv krb5kdc[2857](info): set up 2 sockets
Jun 20 14:59:53 dist-test-slave-n5vv krb5kdc[2857](info): commencing operation
krb5kdc: starting...
W20260620 14:59:54.369396 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.003s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 14:59:54 dist-test-slave-n5vv krb5kdc[2857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 14:59:54.546936 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43817
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:43817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:54.864305 2870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:54.864667 2870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:54.864849 2870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:54.872174 2870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:54.872320 2870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:54.872416 2870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:54.872521 2870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:54.872596 2870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:54.882625 2870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:43817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:54.884344 2870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:54.885839 2870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:54.897563 2875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:54.897617 2878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:54.897629 2876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:54.898372 2870 server_base.cc:1034] running on GCE node
I20260620 14:59:54.899144 2870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:54.900476 2870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:54.901669 2870 hybrid_clock.cc:648] HybridClock initialized: now 1781967594901596 us; error 48 us; skew 500 ppm
I20260620 14:59:54.902058 2870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:54 dist-test-slave-n5vv krb5kdc[2857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967594, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:54.908339 2870 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 14:59:54.910089 2870 webserver.cc:460] Webserver started at http://127.2.70.62:32955/ using document root <none> and password file <none>
I20260620 14:59:54.910797 2870 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:54.910941 2870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:54.911257 2870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:54.914263 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "f2787ea675824c109d22db2e7158dc75"
format_stamp: "Formatted at 2026-06-20 14:59:54 on dist-test-slave-n5vv"
server_key: "4fbc10dd03ac4df53f4c7abcb568d347"
server_key_iv: "2de563be5b4dbb6bd07db8fa08c9b152"
server_key_version: "encryptionkey@0"
I20260620 14:59:54.915017 2870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "f2787ea675824c109d22db2e7158dc75"
format_stamp: "Formatted at 2026-06-20 14:59:54 on dist-test-slave-n5vv"
server_key: "4fbc10dd03ac4df53f4c7abcb568d347"
server_key_iv: "2de563be5b4dbb6bd07db8fa08c9b152"
server_key_version: "encryptionkey@0"
I20260620 14:59:54.920100 2870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 14:59:54.923498 2885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:54.924815 2870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 14:59:54.924999 2870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "f2787ea675824c109d22db2e7158dc75"
format_stamp: "Formatted at 2026-06-20 14:59:54 on dist-test-slave-n5vv"
server_key: "4fbc10dd03ac4df53f4c7abcb568d347"
server_key_iv: "2de563be5b4dbb6bd07db8fa08c9b152"
server_key_version: "encryptionkey@0"
I20260620 14:59:54.925196 2870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:54.945092 2870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:54.949537 2870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:54.949836 2870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:54.967980 2870 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:43817
I20260620 14:59:54.968003 2936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:43817 every 8 connection(s)
I20260620 14:59:54.969410 2870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:54.973102 2937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:54.973585 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 2870
I20260620 14:59:54.973803 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 14:59:54.974448 2328 external_mini_cluster.cc:1428] Setting key 65963af7298667df156650969f42f96d
I20260620 14:59:54.986661 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75: Bootstrap starting.
Jun 20 14:59:54 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 14:59:54.989898 2937 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:54.991240 2937 log.cc:824] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:54.994755 2937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75: No bootstrap required, opened a new log
I20260620 14:59:54.998983 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:54.976563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45390 (local address 127.2.70.62:43817)
0620 14:59:54.977355 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:54.977366 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:54.977410 (+ 44us) server_negotiation.cc:407] Connection header received
0620 14:59:54.979063 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:54.979087 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:54.979475 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:54.979797 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:54.981547 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:54.982936 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:54.984755 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:54.985812 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:54.985987 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:54.990320 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:54.990336 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:54.990357 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:54.990411 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:54.994310 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:54.995094 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:54.995103 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:54.995111 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:54.995324 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:54.995666 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:54.995671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:54.995674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:54.996387 (+ 713us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:54.996674 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:54.997112 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 14:59:54.997590 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":370,"threads_started":1}
I20260620 14:59:55.004936 2937 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2787ea675824c109d22db2e7158dc75" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43817 } }
I20260620 14:59:55.005290 2937 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:55.005362 2937 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2787ea675824c109d22db2e7158dc75, State: Initialized, Role: FOLLOWER
I20260620 14:59:55.006002 2937 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2787ea675824c109d22db2e7158dc75" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43817 } }
I20260620 14:59:55.006208 2937 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:55.006338 2937 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:55.006470 2937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:55.008929 2937 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2787ea675824c109d22db2e7158dc75" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43817 } }
I20260620 14:59:55.009366 2937 leader_election.cc:302] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2787ea675824c109d22db2e7158dc75; no voters:
I20260620 14:59:55.009958 2937 leader_election.cc:288] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:55.010115 2942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:55.010766 2942 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [term 1 LEADER]: Becoming Leader. State: Replica: f2787ea675824c109d22db2e7158dc75, State: Running, Role: LEADER
I20260620 14:59:55.011278 2942 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2787ea675824c109d22db2e7158dc75" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43817 } }
I20260620 14:59:55.012080 2937 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:55.014513 2944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2787ea675824c109d22db2e7158dc75. Latest consensus state: current_term: 1 leader_uuid: "f2787ea675824c109d22db2e7158dc75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2787ea675824c109d22db2e7158dc75" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43817 } } }
I20260620 14:59:55.014521 2943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2787ea675824c109d22db2e7158dc75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2787ea675824c109d22db2e7158dc75" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43817 } } }
I20260620 14:59:55.014876 2944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:55.014941 2943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75 [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:55.016862 2948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:55.024392 2948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:55.033732 2948 catalog_manager.cc:1347] Generated new cluster ID: 06f4fbf6de934e7c96169619ead2a4a0
I20260620 14:59:55.033860 2948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:55.043432 2948 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:55.045125 2948 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:55.052294 2948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2787ea675824c109d22db2e7158dc75: Generated new TSK 0
I20260620 14:59:55.053232 2948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 14:59:55.096223 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43817
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:55.423892 2963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:55.424162 2963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:55.424256 2963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:55.431751 2963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:55.431857 2963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:55.431988 2963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 14:59:55.441776 2963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:55.442991 2963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:55.444239 2963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:55.457347 2969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:55.457312 2971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:55.457598 2968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:55.458724 2963 server_base.cc:1034] running on GCE node
I20260620 14:59:55.459383 2963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:55.460340 2963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:55.461591 2963 hybrid_clock.cc:648] HybridClock initialized: now 1781967595461527 us; error 56 us; skew 500 ppm
I20260620 14:59:55.461915 2963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:55 dist-test-slave-n5vv krb5kdc[2857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:55.465104 2963 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 14:59:55.466711 2963 webserver.cc:460] Webserver started at http://127.2.70.1:46021/ using document root <none> and password file <none>
I20260620 14:59:55.467424 2963 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:55.467574 2963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:55.467881 2963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:55.470791 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "9cb84d95bc784b50857b7c355fdefe4e"
format_stamp: "Formatted at 2026-06-20 14:59:55 on dist-test-slave-n5vv"
server_key: "0d66dff90450bd783c8eb844627fa79d"
server_key_iv: "a0177c73965e6197da159e09c9b68109"
server_key_version: "encryptionkey@0"
I20260620 14:59:55.471606 2963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "9cb84d95bc784b50857b7c355fdefe4e"
format_stamp: "Formatted at 2026-06-20 14:59:55 on dist-test-slave-n5vv"
server_key: "0d66dff90450bd783c8eb844627fa79d"
server_key_iv: "a0177c73965e6197da159e09c9b68109"
server_key_version: "encryptionkey@0"
I20260620 14:59:55.476632 2963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 14:59:55.480037 2978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:55.481392 2963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 14:59:55.481619 2963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "9cb84d95bc784b50857b7c355fdefe4e"
format_stamp: "Formatted at 2026-06-20 14:59:55 on dist-test-slave-n5vv"
server_key: "0d66dff90450bd783c8eb844627fa79d"
server_key_iv: "a0177c73965e6197da159e09c9b68109"
server_key_version: "encryptionkey@0"
I20260620 14:59:55.481827 2963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:55.500600 2963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:55.504830 2963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:55.505127 2963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:55.506109 2963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:55.507537 2963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:55.507638 2963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:55.507787 2963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:55.507858 2963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:55.541533 2963 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:35647
I20260620 14:59:55.541590 3090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:35647 every 8 connection(s)
I20260620 14:59:55.542833 2963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 14:59:55.553136 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 2963
I20260620 14:59:55.553361 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 14:59:55.554005 2328 external_mini_cluster.cc:1428] Setting key 274cf5d32e7a975216a4926e48558db7
Jun 20 14:59:55 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 14:59:55.563206 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:55.545255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:59703 (local address 127.2.70.62:43817)
0620 14:59:55.545728 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:55.545732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:55.546681 (+ 949us) server_negotiation.cc:407] Connection header received
0620 14:59:55.548477 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:55.548482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:55.548577 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:55.548695 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:55.550277 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:55.551002 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:55.552588 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:55.553249 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:55.553393 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:55.557151 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:55.557156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:55.557159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:55.557189 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:55.559727 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:55.560436 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:55.560445 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:55.560449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:55.560615 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:55.561046 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:55.561053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:55.561059 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:55.561470 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:55.561755 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:55.562466 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 14:59:55.562730 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":285,"threads_started":1}
I20260620 14:59:55.563853 3094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:55.545810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43817 (local address 127.2.70.1:59703)
0620 14:59:55.546503 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:55.546545 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:55.548206 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:55.548949 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:55.548957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:55.549468 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:55.550093 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:55.550110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:55.551154 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:55.551161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:55.552438 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:55.552449 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:55.553529 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:55.553538 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:55.553755 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:55.554520 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:55.554557 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:55.556915 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:55.559878 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:55.559885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:55.559899 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:55.560305 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:55.560733 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:55.560736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:55.560738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:55.560911 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:55.561695 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:55.561701 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:55.562031 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 14:59:55.562260 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 14:59:55.562676 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":272,"threads_started":1}
I20260620 14:59:55.565374 3091 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43817
I20260620 14:59:55.565738 3091 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:55.566654 3091 heartbeater.cc:511] Master 127.2.70.62:43817 requested a full tablet report, sending...
I20260620 14:59:55.569072 2902 ts_manager.cc:194] Registered new tserver with Master: 9cb84d95bc784b50857b7c355fdefe4e (127.2.70.1:35647)
I20260620 14:59:55.570658 2902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:59703
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 14:59:55.595216 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43817
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:55.912693 3097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:55.912971 3097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:55.913160 3097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:55.920563 3097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:55.920696 3097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:55.920888 3097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 14:59:55.930955 3097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:55.932224 3097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:55.933568 3097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:55.946367 3103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:55.946473 3102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:55.946444 3105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:55.947649 3097 server_base.cc:1034] running on GCE node
I20260620 14:59:55.948264 3097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:55.949237 3097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:55.950498 3097 hybrid_clock.cc:648] HybridClock initialized: now 1781967595950400 us; error 74 us; skew 500 ppm
I20260620 14:59:55.950824 3097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:55 dist-test-slave-n5vv krb5kdc[2857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:55.954080 3097 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 14:59:55.955606 3097 webserver.cc:460] Webserver started at http://127.2.70.2:39003/ using document root <none> and password file <none>
I20260620 14:59:55.956298 3097 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:55.956440 3097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:55.956751 3097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:55.959697 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76"
format_stamp: "Formatted at 2026-06-20 14:59:55 on dist-test-slave-n5vv"
server_key: "72191eee6a597e65adab77f31ac636a8"
server_key_iv: "c25a9e5d720db2cb39574ce21535b741"
server_key_version: "encryptionkey@0"
I20260620 14:59:55.960551 3097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76"
format_stamp: "Formatted at 2026-06-20 14:59:55 on dist-test-slave-n5vv"
server_key: "72191eee6a597e65adab77f31ac636a8"
server_key_iv: "c25a9e5d720db2cb39574ce21535b741"
server_key_version: "encryptionkey@0"
I20260620 14:59:55.966233 3097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 14:59:55.969820 3112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:55.971185 3097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 14:59:55.971354 3097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76"
format_stamp: "Formatted at 2026-06-20 14:59:55 on dist-test-slave-n5vv"
server_key: "72191eee6a597e65adab77f31ac636a8"
server_key_iv: "c25a9e5d720db2cb39574ce21535b741"
server_key_version: "encryptionkey@0"
I20260620 14:59:55.971575 3097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:55.988828 3097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:55.993213 3097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:55.993572 3097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:55.994527 3097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:55.995798 3097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:55.995898 3097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:55.996050 3097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:55.996124 3097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:56.029260 3097 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:33863
I20260620 14:59:56.029296 3224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:33863 every 8 connection(s)
I20260620 14:59:56.030714 3097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 14:59:56.031586 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 3097
I20260620 14:59:56.031772 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 14:59:56.032301 2328 external_mini_cluster.cc:1428] Setting key 583334c44073544f87815dd930ec1c82
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967595, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 14:59:56.050542 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.033527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:39325 (local address 127.2.70.62:43817)
0620 14:59:56.033701 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.033706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.035056 (+ 1350us) server_negotiation.cc:407] Connection header received
0620 14:59:56.036813 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.036819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.036900 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.037019 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:56.038694 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.039435 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.040972 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.041381 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.041538 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.044942 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:56.044948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:56.044951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:56.045009 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:56.047572 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.048467 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.048472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.048474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.048598 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.048965 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.048968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.048970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.049283 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:56.049401 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.050109 (+ 708us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.050286 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
I20260620 14:59:56.051651 3227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.034155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43817 (local address 127.2.70.2:39325)
0620 14:59:56.034909 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:56.034953 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:56.036559 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:56.037262 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:56.037270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:56.037835 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:56.038509 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.038534 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.039583 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.039587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.040827 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.040838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.041629 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.041639 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.041865 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.042549 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:56.042574 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:56.044718 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:56.047715 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:56.047725 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:56.047743 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:56.048331 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:56.048697 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:56.048700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:56.048702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:56.048867 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:56.049390 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:56.049396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:56.049757 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 14:59:56.049986 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 14:59:56.050425 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":332,"threads_started":1}
I20260620 14:59:56.053295 3225 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43817
I20260620 14:59:56.053664 3225 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:56.054373 3225 heartbeater.cc:511] Master 127.2.70.62:43817 requested a full tablet report, sending...
I20260620 14:59:56.055980 2902 ts_manager.cc:194] Registered new tserver with Master: 89cfc3ad26b54d86b9ca5a6946bf7b76 (127.2.70.2:33863)
I20260620 14:59:56.056751 2902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:39325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 14:59:56.072451 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43817
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:45445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:56.389966 3230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:56.390235 3230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:56.390399 3230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:56.397436 3230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:56.397606 3230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:56.397805 3230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 14:59:56.407430 3230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:45445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:56.408697 3230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:56.410033 3230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:56.423452 3235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:56.423414 3238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:56.423485 3236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:56.424161 3230 server_base.cc:1034] running on GCE node
I20260620 14:59:56.424700 3230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:56.425657 3230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:56.426853 3230 hybrid_clock.cc:648] HybridClock initialized: now 1781967596426794 us; error 53 us; skew 500 ppm
I20260620 14:59:56.427138 3230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:56.430961 3230 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 14:59:56.433024 3230 webserver.cc:460] Webserver started at http://127.2.70.3:34103/ using document root <none> and password file <none>
I20260620 14:59:56.434122 3230 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:56.434257 3230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:56.434676 3230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:56.438778 3230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1"
format_stamp: "Formatted at 2026-06-20 14:59:56 on dist-test-slave-n5vv"
server_key: "02e92470103c016b3a75fe71c9ab44a3"
server_key_iv: "266ed78777c840183f42193807ad41c1"
server_key_version: "encryptionkey@0"
I20260620 14:59:56.439560 3230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1"
format_stamp: "Formatted at 2026-06-20 14:59:56 on dist-test-slave-n5vv"
server_key: "02e92470103c016b3a75fe71c9ab44a3"
server_key_iv: "266ed78777c840183f42193807ad41c1"
server_key_version: "encryptionkey@0"
I20260620 14:59:56.444669 3230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 14:59:56.448244 3245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:56.449944 3230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 14:59:56.450114 3230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1"
format_stamp: "Formatted at 2026-06-20 14:59:56 on dist-test-slave-n5vv"
server_key: "02e92470103c016b3a75fe71c9ab44a3"
server_key_iv: "266ed78777c840183f42193807ad41c1"
server_key_version: "encryptionkey@0"
I20260620 14:59:56.450315 3230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:56.477687 3230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:56.482492 3230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:56.482789 3230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:56.483790 3230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 14:59:56.485080 3230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:56.485152 3230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:56.485330 3230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:56.485384 3230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:56.524364 3230 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:34927
I20260620 14:59:56.524410 3357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:34927 every 8 connection(s)
I20260620 14:59:56.525776 3230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 14:59:56.529160 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 3230
I20260620 14:59:56.529356 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 14:59:56.529815 2328 external_mini_cluster.cc:1428] Setting key 28c30e5a3a162b41105fd45be3816e89
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967596, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 14:59:56.550616 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.528410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:37905 (local address 127.2.70.62:43817)
0620 14:59:56.528620 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.528626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.534170 (+ 5544us) server_negotiation.cc:407] Connection header received
0620 14:59:56.535879 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.535883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.535947 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.536035 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:56.537568 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.538322 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.539835 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.540237 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.540327 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.544231 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:56.544237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:56.544241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:56.544280 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:56.547475 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.548524 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.548529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.548533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.548697 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.549109 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.549114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.549119 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.549410 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:56.549583 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.550209 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.550371 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 14:59:56.551613 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.528954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43817 (local address 127.2.70.3:37905)
0620 14:59:56.534019 (+ 5065us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:56.534058 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:56.535643 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:56.536244 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:56.536252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:56.536790 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:56.537368 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.537384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.538457 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.538460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.539695 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.539706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.540414 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.540418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.540645 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.541355 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:56.541376 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:56.544028 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:56.547633 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:56.547644 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:56.547663 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:56.548339 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:56.548824 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:56.548827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:56.548829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:56.549003 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:56.549589 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:56.549595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:56.549897 (+ 302us) client_negotiation.cc:769] Sending connection context
0620 14:59:56.550126 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 14:59:56.550510 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4808,"thread_start_us":4628,"threads_started":1}
I20260620 14:59:56.553099 3358 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43817
I20260620 14:59:56.553403 3358 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:56.554128 3358 heartbeater.cc:511] Master 127.2.70.62:43817 requested a full tablet report, sending...
I20260620 14:59:56.555541 2902 ts_manager.cc:194] Registered new tserver with Master: a6532fc35d8c4f259da6d7b8e1cdf8f1 (127.2.70.3:34927)
I20260620 14:59:56.556317 2902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:37905
I20260620 14:59:56.557812 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 14:59:56.573933 3091 heartbeater.cc:503] Master 127.2.70.62:43817 was elected leader, sending a full tablet report...
I20260620 14:59:56.576370 3093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.563645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.2.70.62:43817)
0620 14:59:56.563859 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.563867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.564277 (+ 410us) server_negotiation.cc:407] Connection header received
0620 14:59:56.564443 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.564447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.564504 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.564587 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:56.566121 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.566833 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.568012 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.568813 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.568908 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.571135 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:56.571139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:56.571143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:56.571170 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:56.573455 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.574277 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.574284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.574288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.574414 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.574857 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.574862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.574864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.575197 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:56.575447 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.575841 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.576058 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 14:59:56.582615 2902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 14:59:56.589394 2902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 14:59:56.613276 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.601921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:33863 (local address 127.0.0.1:50526)
0620 14:59:56.602934 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:56.602991 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:56.603199 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:56.604396 (+ 1197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:56.604403 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:56.604449 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:56.605765 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.605785 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.608079 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.608086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.610665 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.610679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.612309 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.612313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.612426 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.612455 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:59:56.612608 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 14:59:56.612793 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":67,"spinlock_wait_cycles":8320,"thread_start_us":513,"threads_started":1}
I20260620 14:59:56.613260 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.602109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50526 (local address 127.2.70.2:33863)
0620 14:59:56.603479 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.603489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.603519 (+ 30us) server_negotiation.cc:407] Connection header received
0620 14:59:56.603624 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.603632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.603936 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.604157 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:56.605992 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.607946 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.610844 (+ 2898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.611925 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.612075 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.612211 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.612650 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.612802 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1221,"thread_start_us":1083,"threads_started":1}
I20260620 14:59:56.613834 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.602663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36732 (local address 127.2.70.1:35647)
0620 14:59:56.603830 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.603839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.603878 (+ 39us) server_negotiation.cc:407] Connection header received
0620 14:59:56.603972 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.603979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.604245 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.604436 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:56.606024 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.607499 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.610721 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.612130 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.612223 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.612421 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.613031 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.613207 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":877,"threads_started":1}
I20260620 14:59:56.614310 3371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.602057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35647 (local address 127.0.0.1:36732)
0620 14:59:56.603695 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:56.603714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:56.603820 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:56.604406 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:56.604410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:56.604450 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:56.605857 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.605870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.607626 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.607633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.610550 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.610563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.612629 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.612636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.612804 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.612833 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 14:59:56.613811 (+ 978us) client_negotiation.cc:240] Negotiation successful
0620 14:59:56.613929 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1531,"mutex_wait_us":77,"spinlock_wait_cycles":1536,"thread_start_us":1854,"threads_started":1}
I20260620 14:59:56.615854 3374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.604290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34927 (local address 127.0.0.1:51696)
0620 14:59:56.608006 (+ 3716us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:56.608025 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:56.608138 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:56.608805 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:56.608810 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:56.608840 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:56.609313 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.609325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.611127 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.611133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.613087 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.613103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.615120 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.615126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.615287 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.615320 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 14:59:56.615435 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 14:59:56.615552 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3565,"thread_start_us":3404,"threads_started":1}
I20260620 14:59:56.615929 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.604761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.2.70.3:34927)
0620 14:59:56.605406 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.605416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.608142 (+ 2726us) server_negotiation.cc:407] Connection header received
0620 14:59:56.608305 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.608314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.608546 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.608707 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:56.609524 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.610974 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.613268 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.614932 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.615077 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.615298 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.615481 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.615639 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":293,"threads_started":1}
I20260620 14:59:56.617131 3160 tablet_service.cc:1467] Processing CreateTablet for tablet f1a2db051a9f47cab8696cc899d209fb (DEFAULT_TABLE table=test-table [id=2bcdb12a94bb412282133c7864114e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:59:56.619099 3160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a2db051a9f47cab8696cc899d209fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:56.619656 3293 tablet_service.cc:1467] Processing CreateTablet for tablet f1a2db051a9f47cab8696cc899d209fb (DEFAULT_TABLE table=test-table [id=2bcdb12a94bb412282133c7864114e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:59:56.620894 3293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a2db051a9f47cab8696cc899d209fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:56.620426 3026 tablet_service.cc:1467] Processing CreateTablet for tablet f1a2db051a9f47cab8696cc899d209fb (DEFAULT_TABLE table=test-table [id=2bcdb12a94bb412282133c7864114e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 14:59:56.621810 3026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1a2db051a9f47cab8696cc899d209fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:56.631757 3376 tablet_bootstrap.cc:492] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1: Bootstrap starting.
I20260620 14:59:56.633790 3377 tablet_bootstrap.cc:492] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76: Bootstrap starting.
I20260620 14:59:56.635860 3378 tablet_bootstrap.cc:492] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e: Bootstrap starting.
I20260620 14:59:56.637415 3376 tablet_bootstrap.cc:654] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:56.637588 3377 tablet_bootstrap.cc:654] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:56.638455 3378 tablet_bootstrap.cc:654] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:56.638697 3377 log.cc:824] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:56.638697 3376 log.cc:824] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:56.639547 3378 log.cc:824] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:56.642002 3377 tablet_bootstrap.cc:492] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76: No bootstrap required, opened a new log
I20260620 14:59:56.642310 3376 tablet_bootstrap.cc:492] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1: No bootstrap required, opened a new log
I20260620 14:59:56.642311 3377 ts_tablet_manager.cc:1397] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260620 14:59:56.642561 3376 ts_tablet_manager.cc:1397] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260620 14:59:56.642796 3378 tablet_bootstrap.cc:492] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e: No bootstrap required, opened a new log
I20260620 14:59:56.643040 3378 ts_tablet_manager.cc:1397] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 14:59:56.650008 3378 raft_consensus.cc:348] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.650373 3378 raft_consensus.cc:374] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:56.650457 3378 raft_consensus.cc:729] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb84d95bc784b50857b7c355fdefe4e, State: Initialized, Role: FOLLOWER
I20260620 14:59:56.651048 3378 consensus_queue.cc:260] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.652967 3377 raft_consensus.cc:348] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.652967 3376 raft_consensus.cc:348] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.653396 3376 raft_consensus.cc:374] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:56.653537 3376 raft_consensus.cc:729] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6532fc35d8c4f259da6d7b8e1cdf8f1, State: Initialized, Role: FOLLOWER
I20260620 14:59:56.653396 3377 raft_consensus.cc:374] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:56.653975 3377 raft_consensus.cc:729] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89cfc3ad26b54d86b9ca5a6946bf7b76, State: Initialized, Role: FOLLOWER
I20260620 14:59:56.654398 3376 consensus_queue.cc:260] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.654522 3377 consensus_queue.cc:260] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.656047 3378 ts_tablet_manager.cc:1428] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260620 14:59:56.657475 3358 heartbeater.cc:503] Master 127.2.70.62:43817 was elected leader, sending a full tablet report...
I20260620 14:59:56.658425 3376 ts_tablet_manager.cc:1428] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20260620 14:59:56.659432 3225 heartbeater.cc:503] Master 127.2.70.62:43817 was elected leader, sending a full tablet report...
I20260620 14:59:56.660003 3377 ts_tablet_manager.cc:1428] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76: Time spent starting tablet: real 0.018s user 0.014s sys 0.002s
I20260620 14:59:56.693027 3384 raft_consensus.cc:482] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 14:59:56.693284 3384 raft_consensus.cc:504] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.694929 3384 leader_election.cc:288] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb84d95bc784b50857b7c355fdefe4e (127.2.70.1:35647), a6532fc35d8c4f259da6d7b8e1cdf8f1 (127.2.70.3:34927)
I20260620 14:59:56.703685 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.696517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:50265 (local address 127.2.70.3:34927)
0620 14:59:56.696704 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.696709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.696910 (+ 201us) server_negotiation.cc:407] Connection header received
0620 14:59:56.697137 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.697142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.697205 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.697347 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:56.697911 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.698976 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.701828 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.702773 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.702899 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.703029 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.703316 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.703437 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 14:59:56.703850 3386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.696222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34927 (local address 127.2.70.2:50265)
0620 14:59:56.696816 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:56.696830 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:56.696966 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:56.697361 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:56.697364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:56.697382 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:56.697713 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.697720 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.699119 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.699125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.701680 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.701689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.702967 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.702973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.703159 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.703182 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:59:56.703291 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 14:59:56.703425 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":1,"thread_start_us":316,"threads_started":1}
I20260620 14:59:56.704231 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.695410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35647 (local address 127.2.70.2:56931)
0620 14:59:56.696075 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:56.696096 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:56.696238 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:56.696787 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:56.696791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:56.696824 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 14:59:56.697193 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.697202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.698782 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.698786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.701661 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:56.701677 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.703218 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.703226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:56.703385 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.703408 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 14:59:56.703520 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 14:59:56.703648 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":46,"thread_start_us":319,"threads_started":1}
I20260620 14:59:56.704413 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.695575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:56931 (local address 127.2.70.1:35647)
0620 14:59:56.695754 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.695758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.696189 (+ 431us) server_negotiation.cc:407] Connection header received
0620 14:59:56.696396 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.696402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.696479 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.696606 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 14:59:56.697416 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.698626 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.701888 (+ 3262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.702989 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.703130 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.703237 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.703555 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.703675 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 14:59:56.704922 3313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a2db051a9f47cab8696cc899d209fb" candidate_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" is_pre_election: true
I20260620 14:59:56.705307 3046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a2db051a9f47cab8696cc899d209fb" candidate_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb84d95bc784b50857b7c355fdefe4e" is_pre_election: true
I20260620 14:59:56.705443 3313 raft_consensus.cc:2454] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89cfc3ad26b54d86b9ca5a6946bf7b76 in term 0.
I20260620 14:59:56.705824 3046 raft_consensus.cc:2454] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89cfc3ad26b54d86b9ca5a6946bf7b76 in term 0.
I20260620 14:59:56.706174 3113 leader_election.cc:302] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cfc3ad26b54d86b9ca5a6946bf7b76, a6532fc35d8c4f259da6d7b8e1cdf8f1; no voters:
I20260620 14:59:56.706532 3384 raft_consensus.cc:2781] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 14:59:56.706655 3384 raft_consensus.cc:482] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 14:59:56.706789 3384 raft_consensus.cc:3037] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:56.708986 3384 raft_consensus.cc:504] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.709616 3384 leader_election.cc:288] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [CANDIDATE]: Term 1 election: Requested vote from peers 9cb84d95bc784b50857b7c355fdefe4e (127.2.70.1:35647), a6532fc35d8c4f259da6d7b8e1cdf8f1 (127.2.70.3:34927)
I20260620 14:59:56.710042 3046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a2db051a9f47cab8696cc899d209fb" candidate_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb84d95bc784b50857b7c355fdefe4e"
I20260620 14:59:56.710119 3313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1a2db051a9f47cab8696cc899d209fb" candidate_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1"
I20260620 14:59:56.710250 3046 raft_consensus.cc:3037] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:56.710291 3313 raft_consensus.cc:3037] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:56.713405 3313 raft_consensus.cc:2454] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cfc3ad26b54d86b9ca5a6946bf7b76 in term 1.
I20260620 14:59:56.713405 3046 raft_consensus.cc:2454] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89cfc3ad26b54d86b9ca5a6946bf7b76 in term 1.
I20260620 14:59:56.713799 3113 leader_election.cc:302] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89cfc3ad26b54d86b9ca5a6946bf7b76, a6532fc35d8c4f259da6d7b8e1cdf8f1; no voters:
I20260620 14:59:56.714047 3384 raft_consensus.cc:2781] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:56.714474 3384 raft_consensus.cc:686] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [term 1 LEADER]: Becoming Leader. State: Replica: 89cfc3ad26b54d86b9ca5a6946bf7b76, State: Running, Role: LEADER
I20260620 14:59:56.714926 3384 consensus_queue.cc:237] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } }
I20260620 14:59:56.720104 2900 catalog_manager.cc:5495] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89cfc3ad26b54d86b9ca5a6946bf7b76 (127.2.70.2). New cstate: current_term: 1 leader_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "89cfc3ad26b54d86b9ca5a6946bf7b76" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cb84d95bc784b50857b7c355fdefe4e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6532fc35d8c4f259da6d7b8e1cdf8f1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34927 } health_report { overall_health: UNKNOWN } } }
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 14:59:56.743577 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.728938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.2.70.1:35647)
0620 14:59:56.729133 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.729137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.729690 (+ 553us) server_negotiation.cc:407] Connection header received
0620 14:59:56.729910 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.729914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.729993 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.730079 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:56.731444 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.732189 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.733362 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.733838 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.733928 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.738182 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:56.738195 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:56.738211 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:56.738240 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:56.740838 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.741443 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.741450 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.741452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.741639 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.742035 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.742041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.742045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.742414 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:56.742726 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.743009 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.743222 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 14:59:56.777434 3359 tablet.cc:2369] T f1a2db051a9f47cab8696cc899d209fb P a6532fc35d8c4f259da6d7b8e1cdf8f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 14:59:56.782728 3226 tablet.cc:2369] T f1a2db051a9f47cab8696cc899d209fb P 89cfc3ad26b54d86b9ca5a6946bf7b76: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967596, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 14:59:56.791246 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.776808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.2.70.1:35647)
0620 14:59:56.777048 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.777059 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.777568 (+ 509us) server_negotiation.cc:407] Connection header received
0620 14:59:56.777804 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.777808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.777873 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.777965 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:56.779350 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.780084 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.781342 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.781811 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.781904 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.785925 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:56.785930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:56.785933 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:56.785965 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:56.789028 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.789684 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.789693 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.789696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.789816 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.790146 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.790149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.790151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.790404 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:56.790622 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.790850 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.791011 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 14:59:56.795114 3092 tablet.cc:2369] T f1a2db051a9f47cab8696cc899d209fb P 9cb84d95bc784b50857b7c355fdefe4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 14:59:56.837801 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.822824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.2.70.1:35647)
0620 14:59:56.823024 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.823034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.823367 (+ 333us) server_negotiation.cc:407] Connection header received
0620 14:59:56.823584 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.823588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.823651 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.823755 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:56.825071 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.826075 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.827526 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.828001 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.828147 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.832206 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:56.832211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:56.832214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:56.832255 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:56.835374 (+ 3119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.836056 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.836060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.836062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.836177 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.836565 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.836579 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.836582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.836845 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:56.837015 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.837283 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.837454 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260620 14:59:56.839555 3006 tablet_service.cc:2908] Rejecting scan request for tablet f1a2db051a9f47cab8696cc899d209fb: Uninitialized: clean time has not yet been initialized
I20260620 14:59:56.842849 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 2963
I20260620 14:59:56.855187 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.843837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.2.70.1:35647)
0620 14:59:56.843994 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.843998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.844421 (+ 423us) server_negotiation.cc:407] Connection header received
0620 14:59:56.844554 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.844557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.844622 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.844698 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:56.845969 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.846638 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.847798 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.848188 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.848285 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.850469 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:56.850474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:56.850477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:56.850506 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:56.852767 (+ 2261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.853387 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.853391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.853393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.853562 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.853857 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.853860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.853862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.854264 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:56.854539 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.854758 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.854917 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 14:59:56.855921 3086 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:56.944113 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 2963
I20260620 14:59:56.960999 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3097
Jun 20 14:59:56 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 14:59:56.976044 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:56.961806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.2.70.2:33863)
0620 14:59:56.961989 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:56.961993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:56.962017 (+ 24us) server_negotiation.cc:407] Connection header received
0620 14:59:56.962121 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:56.962125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:56.962183 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:56.962269 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:56.963350 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.964039 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.965229 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:56.965692 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:56.965785 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:56.969801 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:56.969817 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:56.969838 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:56.969876 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:56.973347 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.974019 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.974028 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.974032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.974187 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:56.974516 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:56.974520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:56.974524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:56.974938 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:56.975238 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:56.975478 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 14:59:56.975699 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 14:59:56.976768 3220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:57.066602 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 3097
I20260620 14:59:57.084123 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3230
Jun 20 14:59:57 dist-test-slave-n5vv krb5kdc[2857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 14:59:57.098650 3375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:57.084892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51708 (local address 127.2.70.3:34927)
0620 14:59:57.085065 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:57.085073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:57.085128 (+ 55us) server_negotiation.cc:407] Connection header received
0620 14:59:57.085343 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:57.085347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:57.085417 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:57.085557 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:57.086740 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:57.087448 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:57.088727 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:57.089131 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:57.089264 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:57.093245 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:57.093258 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:57.093274 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:57.093306 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:57.096109 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:57.096745 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:57.096752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:57.096754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:57.096889 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:57.097229 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:57.097232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:57.097234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:57.097634 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:57.097842 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:57.098124 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 14:59:57.098280 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 14:59:57.099247 3353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:57.187678 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 3230
I20260620 14:59:57.203671 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 2870
I20260620 14:59:57.204900 2932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 14:59:57.310799 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 2870
2026-06-20T14:59:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4025 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 14:59:57 dist-test-slave-n5vv krb5kdc[3406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 14:59:57 dist-test-slave-n5vv krb5kdc[3406](info): set up 2 sockets
Jun 20 14:59:57 dist-test-slave-n5vv krb5kdc[3406](info): commencing operation
krb5kdc: starting...
W20260620 14:59:58.393579 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 14:59:58 dist-test-slave-n5vv krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T14:59:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T14:59:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 14:59:58.576435 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42221
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:42221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:58.895417 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:58.895700 3419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:58.895859 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:58.902961 3419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 14:59:58.903069 3419 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 14:59:58.903168 3419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:58.903273 3419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:58.903364 3419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 14:59:58.903475 3419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 14:59:58.913241 3419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:42221
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:58.914703 3419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:58.916080 3419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:58.927982 3424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:58.928068 3427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:58.928104 3425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:58.929219 3419 server_base.cc:1034] running on GCE node
I20260620 14:59:58.930073 3419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:58.931509 3419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:58.932727 3419 hybrid_clock.cc:648] HybridClock initialized: now 1781967598932668 us; error 30 us; skew 500 ppm
I20260620 14:59:58.933063 3419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:58 dist-test-slave-n5vv krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:58.936204 3419 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 14:59:58.937682 3419 webserver.cc:460] Webserver started at http://127.2.70.62:34127/ using document root <none> and password file <none>
I20260620 14:59:58.938371 3419 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:58.938503 3419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:58.938828 3419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:58.941829 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "2f93932b6b8a4e4885eedf3d3989120a"
format_stamp: "Formatted at 2026-06-20 14:59:58 on dist-test-slave-n5vv"
server_key: "8a7d9963e6a80eef1b513a3909b88982"
server_key_iv: "d90b97ed6a625d43623824f92fe03c69"
server_key_version: "encryptionkey@0"
I20260620 14:59:58.942624 3419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "2f93932b6b8a4e4885eedf3d3989120a"
format_stamp: "Formatted at 2026-06-20 14:59:58 on dist-test-slave-n5vv"
server_key: "8a7d9963e6a80eef1b513a3909b88982"
server_key_iv: "d90b97ed6a625d43623824f92fe03c69"
server_key_version: "encryptionkey@0"
I20260620 14:59:58.947667 3419 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260620 14:59:58.951045 3434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:58.952466 3419 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 14:59:58.952651 3419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "2f93932b6b8a4e4885eedf3d3989120a"
format_stamp: "Formatted at 2026-06-20 14:59:58 on dist-test-slave-n5vv"
server_key: "8a7d9963e6a80eef1b513a3909b88982"
server_key_iv: "d90b97ed6a625d43623824f92fe03c69"
server_key_version: "encryptionkey@0"
I20260620 14:59:58.952833 3419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:58.991931 3419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:58.996481 3419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:58.996764 3419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:59.018301 3419 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:42221
I20260620 14:59:59.018419 3495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:42221 every 8 connection(s)
I20260620 14:59:59.019774 3419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 14:59:59.023607 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 3419
I20260620 14:59:59.023849 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 14:59:59.023893 3496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 14:59:59.024690 2328 external_mini_cluster.cc:1428] Setting key a057b349cc8224c5317b10132392a3a8
I20260620 14:59:59.038070 3496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a: Bootstrap starting.
Jun 20 14:59:59 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 14:59:59.042001 3496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a: Neither blocks nor log segments found. Creating new log.
I20260620 14:59:59.043073 3496 log.cc:824] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a: Log is configured to *not* fsync() on all Append() calls
I20260620 14:59:59.045908 3496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a: No bootstrap required, opened a new log
I20260620 14:59:59.048662 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:59.026953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44170 (local address 127.2.70.62:42221)
0620 14:59:59.027826 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:59.027838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:59.027879 (+ 41us) server_negotiation.cc:407] Connection header received
0620 14:59:59.030195 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:59.030218 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:59.030650 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:59.031001 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:59.032695 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.034037 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.035808 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.036455 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.036591 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:59.041154 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:59.041168 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:59.041188 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:59.041232 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:59.044559 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:59.045237 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:59.045251 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:59.045257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:59.045427 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:59.045836 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:59.045845 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:59.045847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:59.046437 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:59.046745 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:59.047182 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 14:59:59.047617 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":427,"threads_started":1}
I20260620 14:59:59.053879 3496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f93932b6b8a4e4885eedf3d3989120a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42221 } }
I20260620 14:59:59.054375 3496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 14:59:59.054487 3496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f93932b6b8a4e4885eedf3d3989120a, State: Initialized, Role: FOLLOWER
I20260620 14:59:59.055173 3496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f93932b6b8a4e4885eedf3d3989120a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42221 } }
I20260620 14:59:59.055437 3496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 14:59:59.055577 3496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 14:59:59.055757 3496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 0 FOLLOWER]: Advancing to term 1
I20260620 14:59:59.058228 3496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f93932b6b8a4e4885eedf3d3989120a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42221 } }
I20260620 14:59:59.058737 3496 leader_election.cc:302] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f93932b6b8a4e4885eedf3d3989120a; no voters:
I20260620 14:59:59.059343 3496 leader_election.cc:288] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 14:59:59.059473 3501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 14:59:59.060230 3501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [term 1 LEADER]: Becoming Leader. State: Replica: 2f93932b6b8a4e4885eedf3d3989120a, State: Running, Role: LEADER
I20260620 14:59:59.060725 3501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f93932b6b8a4e4885eedf3d3989120a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42221 } }
I20260620 14:59:59.061597 3496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 14:59:59.063797 3503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f93932b6b8a4e4885eedf3d3989120a. Latest consensus state: current_term: 1 leader_uuid: "2f93932b6b8a4e4885eedf3d3989120a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f93932b6b8a4e4885eedf3d3989120a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42221 } } }
I20260620 14:59:59.064266 3503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:59.065397 3507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 14:59:59.065397 3502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f93932b6b8a4e4885eedf3d3989120a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f93932b6b8a4e4885eedf3d3989120a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42221 } } }
I20260620 14:59:59.065795 3502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a [sys.catalog]: This master's current role is: LEADER
I20260620 14:59:59.071596 3507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 14:59:59.080233 3507 catalog_manager.cc:1347] Generated new cluster ID: e502f3cdd998400dbd1d9a09d3b62455
I20260620 14:59:59.080355 3507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 14:59:59.118983 3507 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 14:59:59.120339 3507 catalog_manager.cc:1504] Loading token signing keys...
I20260620 14:59:59.130119 3507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a: Generated new TSK 0
I20260620 14:59:59.130990 3507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 14:59:59.180454 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42221
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 14:59:59.527680 3522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 14:59:59.528009 3522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 14:59:59.528187 3522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 14:59:59.535457 3522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 14:59:59.535606 3522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 14:59:59.535753 3522 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 14:59:59.535887 3522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 14:59:59.545811 3522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 14:59:59.547137 3522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 14:59:59.548918 3522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 14:59:59.563251 3527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:59.563557 3530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 14:59:59.563289 3528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 14:59:59.564040 3522 server_base.cc:1034] running on GCE node
I20260620 14:59:59.564719 3522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 14:59:59.565912 3522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 14:59:59.567260 3522 hybrid_clock.cc:648] HybridClock initialized: now 1781967599567157 us; error 71 us; skew 500 ppm
I20260620 14:59:59.567600 3522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 14:59:59 dist-test-slave-n5vv krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 14:59:59.571000 3522 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 14:59:59.572489 3522 webserver.cc:460] Webserver started at http://127.2.70.1:41881/ using document root <none> and password file <none>
I20260620 14:59:59.573203 3522 fs_manager.cc:359] Metadata directory not provided
I20260620 14:59:59.573288 3522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 14:59:59.573635 3522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 14:59:59.576428 3522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "12a3858f936d495b8d816a76b20627eb"
format_stamp: "Formatted at 2026-06-20 14:59:59 on dist-test-slave-n5vv"
server_key: "ebe45967fc3476e4f57ef6d55479b8be"
server_key_iv: "d74950d32fe19ffd12e274f07fb68639"
server_key_version: "encryptionkey@0"
I20260620 14:59:59.577175 3522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "12a3858f936d495b8d816a76b20627eb"
format_stamp: "Formatted at 2026-06-20 14:59:59 on dist-test-slave-n5vv"
server_key: "ebe45967fc3476e4f57ef6d55479b8be"
server_key_iv: "d74950d32fe19ffd12e274f07fb68639"
server_key_version: "encryptionkey@0"
I20260620 14:59:59.582212 3522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 14:59:59.585440 3537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:59.586969 3522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 14:59:59.587116 3522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "12a3858f936d495b8d816a76b20627eb"
format_stamp: "Formatted at 2026-06-20 14:59:59 on dist-test-slave-n5vv"
server_key: "ebe45967fc3476e4f57ef6d55479b8be"
server_key_iv: "d74950d32fe19ffd12e274f07fb68639"
server_key_version: "encryptionkey@0"
I20260620 14:59:59.587311 3522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 14:59:59.622748 3522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 14:59:59.627236 3522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 14:59:59.627460 3522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 14:59:59.630514 3522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 14:59:59.630627 3522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 14:59:59.630774 3522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 14:59:59.630846 3522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 14:59:59 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967599, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 14:59:59.652621 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:59.630843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:57753 (local address 127.2.70.62:42221)
0620 14:59:59.631320 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:59.631329 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:59.632561 (+ 1232us) server_negotiation.cc:407] Connection header received
0620 14:59:59.634620 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:59.634625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:59.634719 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:59.634836 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:59.637128 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.638420 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.640969 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.641733 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.641887 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:59.645663 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:59.645688 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:59.645694 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:59.645739 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:59.648976 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:59.649992 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:59.649997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:59.650001 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:59.650148 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:59.650743 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:59.650749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:59.650754 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:59.651188 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:59.651342 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:59.652054 (+ 712us) server_negotiation.cc:299] Negotiation successful
0620 14:59:59.652263 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":247,"threads_started":1}
I20260620 14:59:59.653905 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:59.631537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42221 (local address 127.2.70.1:57753)
0620 14:59:59.632377 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:59.632420 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:59.634338 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:59.635112 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:59.635127 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:59.635985 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:59.636839 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:59.636862 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.638632 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.638640 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:59.640794 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:59.640814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.642030 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.642043 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:59.642262 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:59.643005 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:59.643027 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:59.645391 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:59.649121 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:59.649133 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:59.649155 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:59.649851 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:59.650300 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:59.650308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:59.650313 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:59.650610 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:59.651326 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:59.651334 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:59.651698 (+ 364us) client_negotiation.cc:769] Sending connection context
0620 14:59:59.651952 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 14:59:59.652485 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":315,"threads_started":1}
I20260620 14:59:59.676573 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:59.661506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:44931 (local address 127.2.70.62:42221)
0620 14:59:59.661699 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 14:59:59.661707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 14:59:59.662384 (+ 677us) server_negotiation.cc:407] Connection header received
0620 14:59:59.662724 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 14:59:59.662733 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 14:59:59.662846 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 14:59:59.662978 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 14:59:59.664212 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.665231 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.667174 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.667667 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.667805 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:59.670146 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 14:59:59.670154 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 14:59:59.670162 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 14:59:59.670205 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 14:59:59.673261 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:59.674382 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:59.674388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:59.674391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:59.674536 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 14:59:59.675080 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 14:59:59.675084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 14:59:59.675086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 14:59:59.675364 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 14:59:59.675483 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 14:59:59.676056 (+ 573us) server_negotiation.cc:299] Negotiation successful
0620 14:59:59.676256 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 14:59:59.677079 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 14:59:59.661456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42221 (local address 127.2.70.1:44931)
0620 14:59:59.662272 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0620 14:59:59.662302 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 14:59:59.662505 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 14:59:59.663011 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 14:59:59.663015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 14:59:59.663516 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0620 14:59:59.664054 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:59.664067 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.665363 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.665372 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:59.666991 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 14:59:59.667003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 14:59:59.667853 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 14:59:59.667859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 14:59:59.668020 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 14:59:59.668852 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 14:59:59.668871 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 14:59:59.669901 (+ 1030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 14:59:59.673417 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:59.673425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:59.673445 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:59.674223 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:59.674654 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 14:59:59.674662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 14:59:59.674669 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 14:59:59.674867 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 14:59:59.675493 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 14:59:59.675501 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 14:59:59.675759 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 14:59:59.675931 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 14:59:59.676402 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":375,"threads_started":1}
I20260620 14:59:59.685277 3522 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:45183
I20260620 14:59:59.685293 3657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:45183 every 8 connection(s)
I20260620 14:59:59.686534 3522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 14:59:59.688663 3658 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42221
I20260620 14:59:59.688891 3658 heartbeater.cc:464] Registering TS with master...
I20260620 14:59:59.689599 3658 heartbeater.cc:511] Master 127.2.70.62:42221 requested a full tablet report, sending...
I20260620 14:59:59.690388 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 3522
I20260620 14:59:59.690665 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 14:59:59.691331 2328 external_mini_cluster.cc:1428] Setting key c1ce734dd61e5ccedf54dcff7e539294
I20260620 14:59:59.691756 3451 ts_manager.cc:194] Registered new tserver with Master: 12a3858f936d495b8d816a76b20627eb (127.2.70.1:45183)
I20260620 14:59:59.693691 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:57753
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 14:59:59.732394 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42221
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:00.053473 3662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:00.053843 3662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:00.054041 3662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:00.062165 3662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:00.062282 3662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:00.062444 3662 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:00:00.062568 3662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:00.072608 3662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:00.073947 3662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:00.075217 3662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:00.088258 3670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:00.088366 3667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:00.088379 3668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:00.089349 3662 server_base.cc:1034] running on GCE node
I20260620 15:00:00.090065 3662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:00.091332 3662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:00.092597 3662 hybrid_clock.cc:648] HybridClock initialized: now 1781967600092503 us; error 75 us; skew 500 ppm
I20260620 15:00:00.092943 3662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:00 dist-test-slave-n5vv krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:00.099048 3662 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:00:00.100530 3662 webserver.cc:460] Webserver started at http://127.2.70.2:38605/ using document root <none> and password file <none>
I20260620 15:00:00.101259 3662 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:00.101398 3662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:00.101789 3662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:00.104830 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "bdcc2f08341043d0b96242b7deeeb888"
format_stamp: "Formatted at 2026-06-20 15:00:00 on dist-test-slave-n5vv"
server_key: "779523591ef9481f3229a9ae02702914"
server_key_iv: "9ed902dfe03135f8155121c53a68bc96"
server_key_version: "encryptionkey@0"
I20260620 15:00:00.105789 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "bdcc2f08341043d0b96242b7deeeb888"
format_stamp: "Formatted at 2026-06-20 15:00:00 on dist-test-slave-n5vv"
server_key: "779523591ef9481f3229a9ae02702914"
server_key_iv: "9ed902dfe03135f8155121c53a68bc96"
server_key_version: "encryptionkey@0"
I20260620 15:00:00.111209 3662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:00.114737 3677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:00.116047 3662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:00.116232 3662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "bdcc2f08341043d0b96242b7deeeb888"
format_stamp: "Formatted at 2026-06-20 15:00:00 on dist-test-slave-n5vv"
server_key: "779523591ef9481f3229a9ae02702914"
server_key_iv: "9ed902dfe03135f8155121c53a68bc96"
server_key_version: "encryptionkey@0"
I20260620 15:00:00.116463 3662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:00.141106 3662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:00.145222 3662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:00.145534 3662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:00.148191 3662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:00.148322 3662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:00.148469 3662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:00.148543 3662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:00:00 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:00.173784 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.149041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:53817 (local address 127.2.70.62:42221)
0620 15:00:00.149223 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:00.149236 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:00.150789 (+ 1553us) server_negotiation.cc:407] Connection header received
0620 15:00:00.152885 (+ 2096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:00.152891 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:00.152988 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:00.153097 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:00.155832 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.157049 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.159848 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.160365 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.160513 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.166099 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:00.166107 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:00.166114 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:00.166157 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:00.169964 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.171004 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.171012 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.171017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.171173 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.171781 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.171785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.171787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.172199 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:00.172390 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:00.173151 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 15:00:00.173387 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:00:00.175770 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.149781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42221 (local address 127.2.70.2:53817)
0620 15:00:00.150629 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:00.150673 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:00.152604 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:00.153423 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:00.153443 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:00.154488 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:00.155606 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.155647 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.157254 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.157258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.159691 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.159704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.160649 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.160659 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.161039 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.161978 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:00.162005 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:00.165756 (+ 3751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:00.170112 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:00.170121 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:00.170144 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:00.170818 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:00.171293 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:00.171298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:00.171302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:00.171617 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:00.172359 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:00.172369 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:00.172774 (+ 405us) client_negotiation.cc:769] Sending connection context
0620 15:00:00.173055 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 15:00:00.173718 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":373,"threads_started":1}
I20260620 15:00:00.197911 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.181730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:38927 (local address 127.2.70.62:42221)
0620 15:00:00.181903 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:00.181907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:00.182558 (+ 651us) server_negotiation.cc:407] Connection header received
0620 15:00:00.182791 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:00.182796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:00.182872 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:00.182975 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:00.184424 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.185544 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.187762 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.188329 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.188477 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.190919 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:00.190929 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:00.190936 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:00.190979 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:00.194350 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.195309 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.195316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.195322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.195451 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.196001 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.196010 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.196016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.196544 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:00.196720 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:00.197247 (+ 527us) server_negotiation.cc:299] Negotiation successful
0620 15:00:00.197538 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:00:00.198066 3754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.181747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42221 (local address 127.2.70.2:38927)
0620 15:00:00.182432 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:00.182448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:00.182640 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:00.183001 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:00.183009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:00.183759 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:00.184265 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.184279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.185703 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.185708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.187545 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.187567 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.188532 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.188538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.188714 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.189414 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:00.189442 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:00.190653 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:00.194504 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:00.194510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:00.194514 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:00.195154 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:00.195586 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:00.195591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:00.195595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:00.195802 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:00.196669 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:00.196677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:00.196943 (+ 266us) client_negotiation.cc:769] Sending connection context
0620 15:00:00.197154 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 15:00:00.197543 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":433,"threads_started":1}
I20260620 15:00:00.200516 3662 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:36057
I20260620 15:00:00.200565 3796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:36057 every 8 connection(s)
I20260620 15:00:00.202049 3662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:00.204098 3797 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42221
I20260620 15:00:00.204301 3797 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:00.204908 3797 heartbeater.cc:511] Master 127.2.70.62:42221 requested a full tablet report, sending...
I20260620 15:00:00.206256 3451 ts_manager.cc:194] Registered new tserver with Master: bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057)
I20260620 15:00:00.207055 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:53817
I20260620 15:00:00.210134 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 3662
I20260620 15:00:00.210292 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:00.210688 2328 external_mini_cluster.cc:1428] Setting key 5dbf097334d3623518038384285a033e
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:00:00.250252 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42221
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:00.579480 3801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:00.579825 3801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:00.579996 3801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:00.587457 3801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:00.587628 3801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:00.587772 3801 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:00:00.587908 3801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:00.598522 3801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:42221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:00.599846 3801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:00.601786 3801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:00.615046 3807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:00.614996 3809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:00.615094 3806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:00.619419 3801 server_base.cc:1034] running on GCE node
I20260620 15:00:00.620218 3801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:00.621380 3801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:00.622630 3801 hybrid_clock.cc:648] HybridClock initialized: now 1781967600622561 us; error 45 us; skew 500 ppm
I20260620 15:00:00.622946 3801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:00 dist-test-slave-n5vv krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:00.626251 3801 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:00:00.627831 3801 webserver.cc:460] Webserver started at http://127.2.70.3:42713/ using document root <none> and password file <none>
I20260620 15:00:00.628562 3801 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:00.628690 3801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:00.629019 3801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:00.631971 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "cbc4c0ab4a2843769b004cec92ae84c0"
format_stamp: "Formatted at 2026-06-20 15:00:00 on dist-test-slave-n5vv"
server_key: "c3c45f2293e58497e598fe5e7f199278"
server_key_iv: "4ea97c77d3ee4afc77fa901a65102ab1"
server_key_version: "encryptionkey@0"
I20260620 15:00:00.632786 3801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "cbc4c0ab4a2843769b004cec92ae84c0"
format_stamp: "Formatted at 2026-06-20 15:00:00 on dist-test-slave-n5vv"
server_key: "c3c45f2293e58497e598fe5e7f199278"
server_key_iv: "4ea97c77d3ee4afc77fa901a65102ab1"
server_key_version: "encryptionkey@0"
I20260620 15:00:00.638003 3801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 15:00:00.641259 3816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:00.642604 3801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:00.642769 3801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "cbc4c0ab4a2843769b004cec92ae84c0"
format_stamp: "Formatted at 2026-06-20 15:00:00 on dist-test-slave-n5vv"
server_key: "c3c45f2293e58497e598fe5e7f199278"
server_key_iv: "4ea97c77d3ee4afc77fa901a65102ab1"
server_key_version: "encryptionkey@0"
I20260620 15:00:00.642987 3801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:00.664441 3801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:00.668881 3801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:00.669155 3801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:00.671886 3801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:00.671999 3801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:00.672183 3801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:00.672272 3801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:00:00 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:00.695634 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.672396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:53131 (local address 127.2.70.62:42221)
0620 15:00:00.672638 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:00.672648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:00.674214 (+ 1566us) server_negotiation.cc:407] Connection header received
0620 15:00:00.676289 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:00.676295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:00.676431 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:00.676563 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:00.679090 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.680134 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.681926 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.682325 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.682436 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.687971 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:00.687979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:00.687983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:00.688024 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:00.691413 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.692407 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.692414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.692418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.692578 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.693247 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.693255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.693261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.693731 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:00.693906 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:00.695058 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0620 15:00:00.695274 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:00:00.696501 3658 heartbeater.cc:503] Master 127.2.70.62:42221 was elected leader, sending a full tablet report...
I20260620 15:00:00.696956 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.673086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42221 (local address 127.2.70.3:53131)
0620 15:00:00.674032 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:00.674087 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:00.675914 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:00.676925 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:00.676940 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:00.677878 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:00.678835 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.678858 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.680289 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.680295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.681765 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.681779 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.682712 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.682726 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.683106 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.683922 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:00.683955 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:00.687660 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:00.691580 (+ 3920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:00.691589 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:00.691605 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:00.692258 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:00.692702 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:00.692708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:00.692712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:00.693065 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:00.693996 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:00.694007 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:00.694569 (+ 562us) client_negotiation.cc:769] Sending connection context
0620 15:00:00.694977 (+ 408us) client_negotiation.cc:240] Negotiation successful
0620 15:00:00.695509 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":417,"threads_started":1}
I20260620 15:00:00.726280 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.705540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:57131 (local address 127.2.70.62:42221)
0620 15:00:00.705751 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:00.705761 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:00.710191 (+ 4430us) server_negotiation.cc:407] Connection header received
0620 15:00:00.710452 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:00.710457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:00.710547 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:00.710670 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:00.712545 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.713634 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.716437 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.716906 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.717008 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.719862 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:00.719869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:00.719874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:00.719919 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:00.723378 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.724244 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.724251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.724254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.724396 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.724764 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.724770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.724774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.725043 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:00.725196 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:00.725692 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 15:00:00.725912 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:00:00.726383 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.705442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42221 (local address 127.2.70.3:57131)
0620 15:00:00.710057 (+ 4615us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:00.710089 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:00.710278 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:00.710738 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:00.710745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:00.711610 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:00.712294 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.712319 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.713819 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.713829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.716233 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.716259 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.717144 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.717151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.717346 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.718221 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:00.718239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:00.719550 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:00.723520 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:00.723526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:00.723530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:00.724111 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:00.724524 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:00.724530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:00.724533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:00.724658 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:00.725162 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:00.725167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:00.725377 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 15:00:00.725558 (+ 181us) client_negotiation.cc:240] Negotiation successful
0620 15:00:00.725913 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4456,"thread_start_us":4255,"threads_started":1}
I20260620 15:00:00.727598 3801 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:43373
I20260620 15:00:00.727622 3935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:43373 every 8 connection(s)
I20260620 15:00:00.728871 3801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:00.731212 3936 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42221
I20260620 15:00:00.731444 3936 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:00.732075 3936 heartbeater.cc:511] Master 127.2.70.62:42221 requested a full tablet report, sending...
I20260620 15:00:00.733404 3451 ts_manager.cc:194] Registered new tserver with Master: cbc4c0ab4a2843769b004cec92ae84c0 (127.2.70.3:43373)
I20260620 15:00:00.734259 3451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:53131
I20260620 15:00:00.738250 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 3801
I20260620 15:00:00.738440 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:00.738862 2328 external_mini_cluster.cc:1428] Setting key e9ee7508b9cfaebdcfb2d4745533b852
I20260620 15:00:00.741608 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:00:00 dist-test-slave-n5vv krb5kdc[3406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:00:00 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:00.786918 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.773712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44184 (local address 127.2.70.62:42221)
0620 15:00:00.773940 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:00.773948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:00.774111 (+ 163us) server_negotiation.cc:407] Connection header received
0620 15:00:00.774258 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:00.774262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:00.774397 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:00.774498 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:00.775825 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.776522 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.777935 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.778345 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.778437 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.782066 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:00.782071 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:00.782074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:00.782104 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:00.784665 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.785222 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.785226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.785229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.785344 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:00.785721 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:00.785724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:00.785726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:00.785965 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:00.786137 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:00.786361 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 15:00:00.786534 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 15:00:00.791064 3451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:00.797843 3451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:00.832487 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.822930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40526 (local address 127.2.70.2:36057)
0620 15:00:00.824140 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:00.824148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:00.824785 (+ 637us) server_negotiation.cc:407] Connection header received
0620 15:00:00.824876 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:00.824882 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:00.825139 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:00.825321 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:00.826132 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.827455 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.830047 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.831069 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.831189 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.831400 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:00.832025 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 15:00:00.832183 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1061,"thread_start_us":929,"threads_started":1}
I20260620 15:00:00.832551 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.822713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:36057 (local address 127.0.0.1:40526)
0620 15:00:00.824536 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:00.824552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:00.824661 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:00.825555 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:00.825560 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:00.825582 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:00.825935 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.825946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.827619 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.827626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.829897 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.829911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.831554 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.831562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.831779 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.831831 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 15:00:00.831979 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 15:00:00.832154 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"mutex_wait_us":3,"thread_start_us":1568,"threads_started":1}
I20260620 15:00:00.834223 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.810199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.2.70.1:45183)
0620 15:00:00.822548 (+ 12349us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:00.822563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:00.823701 (+ 1138us) server_negotiation.cc:407] Connection header received
0620 15:00:00.824030 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:00.824039 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:00.824331 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:00.824529 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:00.825557 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.827158 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.831455 (+ 4297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.833176 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.833306 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.833521 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:00.833804 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 15:00:00.833958 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12123,"thread_start_us":12378,"threads_started":1}
I20260620 15:00:00.834475 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.810025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:45183 (local address 127.0.0.1:55516)
0620 15:00:00.823575 (+ 13550us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:00.823593 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:00.823811 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:00.824830 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:00.824837 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:00.824865 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:00.825293 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.825307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.827317 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.827321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.831258 (+ 3937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.831274 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.833374 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.833380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.833558 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.833593 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:00:00.833759 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 15:00:00.833884 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13424,"mutex_wait_us":1453,"thread_start_us":11497,"threads_started":1}
I20260620 15:00:00.835274 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.810062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.2.70.3:43373)
0620 15:00:00.823378 (+ 13316us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:00.823388 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:00.824112 (+ 724us) server_negotiation.cc:407] Connection header received
0620 15:00:00.824629 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:00.824635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:00.824825 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:00.825015 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:00.826152 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.829557 (+ 3405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.832049 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.833779 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.833904 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.834111 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:00.834737 (+ 626us) server_negotiation.cc:299] Negotiation successful
0620 15:00:00.834928 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13133,"thread_start_us":12913,"threads_started":1}
I20260620 15:00:00.836094 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:00.809829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:43373 (local address 127.0.0.1:49848)
0620 15:00:00.823297 (+ 13468us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:00.823353 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:00.823642 (+ 289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:00.825014 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:00.825018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:00.825040 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:00.825575 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.825588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.829692 (+ 4104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.829699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.831862 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:00.831876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:00.834265 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:00.834273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:00.834433 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:00.834462 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:00:00.834573 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 15:00:00.835802 (+ 1229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13115,"mutex_wait_us":1,"thread_start_us":13590,"threads_started":1}
I20260620 15:00:00.836711 3727 tablet_service.cc:1467] Processing CreateTablet for tablet c17261a4644d4b18847a2dc68b63fa3f (DEFAULT_TABLE table=test-table [id=f81f7aaf3c4a43e1a36d3b79f95964d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:00.838232 3727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c17261a4644d4b18847a2dc68b63fa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:00.837888 3588 tablet_service.cc:1467] Processing CreateTablet for tablet c17261a4644d4b18847a2dc68b63fa3f (DEFAULT_TABLE table=test-table [id=f81f7aaf3c4a43e1a36d3b79f95964d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:00.839174 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c17261a4644d4b18847a2dc68b63fa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:00.838866 3866 tablet_service.cc:1467] Processing CreateTablet for tablet c17261a4644d4b18847a2dc68b63fa3f (DEFAULT_TABLE table=test-table [id=f81f7aaf3c4a43e1a36d3b79f95964d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:00.839991 3866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c17261a4644d4b18847a2dc68b63fa3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:00.852660 3955 tablet_bootstrap.cc:492] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Bootstrap starting.
I20260620 15:00:00.856720 3955 tablet_bootstrap.cc:654] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:00.858048 3955 log.cc:824] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:00.859045 3956 tablet_bootstrap.cc:492] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Bootstrap starting.
I20260620 15:00:00.863911 3956 tablet_bootstrap.cc:654] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:00.865228 3957 tablet_bootstrap.cc:492] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Bootstrap starting.
I20260620 15:00:00.865967 3956 log.cc:824] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:00.867762 3955 tablet_bootstrap.cc:492] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: No bootstrap required, opened a new log
I20260620 15:00:00.868045 3955 ts_tablet_manager.cc:1397] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20260620 15:00:00.869256 3956 tablet_bootstrap.cc:492] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: No bootstrap required, opened a new log
I20260620 15:00:00.869387 3957 tablet_bootstrap.cc:654] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:00.869585 3956 ts_tablet_manager.cc:1397] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260620 15:00:00.870747 3957 log.cc:824] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:00.873883 3957 tablet_bootstrap.cc:492] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: No bootstrap required, opened a new log
I20260620 15:00:00.874147 3957 ts_tablet_manager.cc:1397] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 15:00:00.878136 3956 raft_consensus.cc:348] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:00.878556 3956 raft_consensus.cc:374] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:00.878648 3956 raft_consensus.cc:729] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a3858f936d495b8d816a76b20627eb, State: Initialized, Role: FOLLOWER
I20260620 15:00:00.878574 3955 raft_consensus.cc:348] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:00.878983 3955 raft_consensus.cc:374] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:00.879081 3955 raft_consensus.cc:729] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc4c0ab4a2843769b004cec92ae84c0, State: Initialized, Role: FOLLOWER
I20260620 15:00:00.879465 3956 consensus_queue.cc:260] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:00.879650 3955 consensus_queue.cc:260] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:00.881287 3956 ts_tablet_manager.cc:1428] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260620 15:00:00.882206 3936 heartbeater.cc:503] Master 127.2.70.62:42221 was elected leader, sending a full tablet report...
I20260620 15:00:00.882802 3955 ts_tablet_manager.cc:1428] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260620 15:00:00.885172 3957 raft_consensus.cc:348] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:00.885612 3957 raft_consensus.cc:374] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:00.885712 3957 raft_consensus.cc:729] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcc2f08341043d0b96242b7deeeb888, State: Initialized, Role: FOLLOWER
I20260620 15:00:00.886298 3957 consensus_queue.cc:260] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:00.887871 3797 heartbeater.cc:503] Master 127.2.70.62:42221 was elected leader, sending a full tablet report...
I20260620 15:00:00.889406 3957 ts_tablet_manager.cc:1428] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260620 15:00:00.939407 3659 tablet.cc:2369] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:00:00.954156 3798 tablet.cc:2369] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:00:00.980943 3937 tablet.cc:2369] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:01.025995 3961 raft_consensus.cc:482] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:01.026341 3961 raft_consensus.cc:504] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:01.028864 3961 leader_election.cc:288] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbc4c0ab4a2843769b004cec92ae84c0 (127.2.70.3:43373), bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057)
I20260620 15:00:01.036404 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.029036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:59275 (local address 127.2.70.2:36057)
0620 15:00:01.029238 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.029247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.030048 (+ 801us) server_negotiation.cc:407] Connection header received
0620 15:00:01.030373 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.030379 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.030451 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.030546 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:01.031196 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.032018 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.034480 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.035608 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.035748 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.035856 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.036076 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.036200 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:00:01.036597 3965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.028718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:36057 (local address 127.2.70.1:59275)
0620 15:00:01.029936 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:01.029956 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:01.030190 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:01.030585 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:01.030590 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:01.030619 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:01.031026 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.031038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.032154 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.032160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.034330 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.034343 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.035814 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.035818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.035921 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.035944 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:00:01.036019 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 15:00:01.036174 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1066,"mutex_wait_us":5,"thread_start_us":883,"threads_started":1}
I20260620 15:00:01.037029 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.029386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:60777 (local address 127.2.70.3:43373)
0620 15:00:01.029588 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.029595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.030337 (+ 742us) server_negotiation.cc:407] Connection header received
0620 15:00:01.030603 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.030608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.030689 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.030747 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:01.031486 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.032523 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.034758 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.035963 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.036123 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.036219 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.036583 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.036788 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:00:01.037640 3747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17261a4644d4b18847a2dc68b63fa3f" candidate_uuid: "12a3858f936d495b8d816a76b20627eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcc2f08341043d0b96242b7deeeb888" is_pre_election: true
I20260620 15:00:01.037992 3964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.029088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:43373 (local address 127.2.70.1:60777)
0620 15:00:01.030217 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:01.030232 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:01.030431 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:01.030902 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:01.030905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:01.030923 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:01.031328 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.031339 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.032672 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.032677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.034585 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.034594 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.036210 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.036216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.036389 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.036418 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 15:00:01.037067 (+ 649us) client_negotiation.cc:240] Negotiation successful
0620 15:00:01.037216 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":28,"thread_start_us":882,"threads_started":1}
I20260620 15:00:01.038177 3747 raft_consensus.cc:2454] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a3858f936d495b8d816a76b20627eb in term 0.
I20260620 15:00:01.038898 3540 leader_election.cc:302] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3858f936d495b8d816a76b20627eb, bdcc2f08341043d0b96242b7deeeb888; no voters:
I20260620 15:00:01.039057 3891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17261a4644d4b18847a2dc68b63fa3f" candidate_uuid: "12a3858f936d495b8d816a76b20627eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" is_pre_election: true
I20260620 15:00:01.039350 3961 raft_consensus.cc:2781] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:01.039516 3961 raft_consensus.cc:482] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:01.039635 3961 raft_consensus.cc:3037] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.039606 3891 raft_consensus.cc:2454] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a3858f936d495b8d816a76b20627eb in term 0.
I20260620 15:00:01.041960 3961 raft_consensus.cc:504] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:01.042555 3961 leader_election.cc:288] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [CANDIDATE]: Term 1 election: Requested vote from peers cbc4c0ab4a2843769b004cec92ae84c0 (127.2.70.3:43373), bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057)
I20260620 15:00:01.042955 3891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17261a4644d4b18847a2dc68b63fa3f" candidate_uuid: "12a3858f936d495b8d816a76b20627eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc4c0ab4a2843769b004cec92ae84c0"
I20260620 15:00:01.042989 3747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c17261a4644d4b18847a2dc68b63fa3f" candidate_uuid: "12a3858f936d495b8d816a76b20627eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcc2f08341043d0b96242b7deeeb888"
I20260620 15:00:01.043156 3891 raft_consensus.cc:3037] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.043171 3747 raft_consensus.cc:3037] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.045207 3891 raft_consensus.cc:2454] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a3858f936d495b8d816a76b20627eb in term 1.
I20260620 15:00:01.045250 3747 raft_consensus.cc:2454] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a3858f936d495b8d816a76b20627eb in term 1.
I20260620 15:00:01.045637 3540 leader_election.cc:302] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3858f936d495b8d816a76b20627eb, bdcc2f08341043d0b96242b7deeeb888; no voters:
I20260620 15:00:01.045955 3961 raft_consensus.cc:2781] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:01.046393 3961 raft_consensus.cc:686] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 1 LEADER]: Becoming Leader. State: Replica: 12a3858f936d495b8d816a76b20627eb, State: Running, Role: LEADER
I20260620 15:00:01.046939 3961 consensus_queue.cc:237] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } }
I20260620 15:00:01.051213 3451 catalog_manager.cc:5495] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 12a3858f936d495b8d816a76b20627eb (127.2.70.1). New cstate: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:01.077689 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.072891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.2.70.1:45183)
0620 15:00:01.073058 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.073062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.073079 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:00:01.073242 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.073245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.073315 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.073428 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:01.073794 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.074461 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.075717 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.076105 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.076192 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.076607 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:01.077093 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:01.077219 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.077331 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.077390 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:00:01.085378 3891 raft_consensus.cc:1261] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Refusing update from remote peer 12a3858f936d495b8d816a76b20627eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:01.085379 3747 raft_consensus.cc:1261] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Refusing update from remote peer 12a3858f936d495b8d816a76b20627eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:01.086493 3966 consensus_queue.cc:1035] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:01.086875 3961 consensus_queue.cc:1035] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:01.103394 3971 mvcc.cc:205] Tried to move back new op lower bound from 7298939294035623936 to 7298939293892141056. Current Snapshot: MvccSnapshot[applied={T|T < 7298939294035623936}]
I20260620 15:00:01.120947 3451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:01.121255 3451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:01.124866 3451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a: Sending DeleteTablet for 3 replicas of tablet c17261a4644d4b18847a2dc68b63fa3f
I20260620 15:00:01.126067 3866 tablet_service.cc:1514] Processing DeleteTablet for tablet c17261a4644d4b18847a2dc68b63fa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:01 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:49848
I20260620 15:00:01.126124 3727 tablet_service.cc:1514] Processing DeleteTablet for tablet c17261a4644d4b18847a2dc68b63fa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:01 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:40526
I20260620 15:00:01.126130 3588 tablet_service.cc:1514] Processing DeleteTablet for tablet c17261a4644d4b18847a2dc68b63fa3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:01 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:55516
I20260620 15:00:01.128930 3451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:01.130476 3451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:01.134085 3979 tablet_replica.cc:330] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: stopping tablet replica
I20260620 15:00:01.134637 3979 raft_consensus.cc:2229] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:01.135243 3979 raft_consensus.cc:2258] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:01.139108 3978 tablet_replica.cc:330] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: stopping tablet replica
I20260620 15:00:01.139514 3978 raft_consensus.cc:2229] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:01.139701 3979 ts_tablet_manager.cc:1905] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:01.139972 3978 raft_consensus.cc:2258] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:01.142196 3978 ts_tablet_manager.cc:1905] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:01.143088 3726 tablet_service.cc:1467] Processing CreateTablet for tablet 61c413fd84d74e50a9c29f7c671e551a (DEFAULT_TABLE table=test-table [id=f0c03b1a16a746c38cede509e80b91b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:01.143545 3866 tablet_service.cc:1467] Processing CreateTablet for tablet 61c413fd84d74e50a9c29f7c671e551a (DEFAULT_TABLE table=test-table [id=f0c03b1a16a746c38cede509e80b91b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:01.143936 3866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c413fd84d74e50a9c29f7c671e551a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:01.144086 3726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c413fd84d74e50a9c29f7c671e551a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:01.144030 3588 tablet_service.cc:1467] Processing CreateTablet for tablet 61c413fd84d74e50a9c29f7c671e551a (DEFAULT_TABLE table=test-table [id=f0c03b1a16a746c38cede509e80b91b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:01.144438 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c413fd84d74e50a9c29f7c671e551a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:01.152614 3957 tablet_bootstrap.cc:492] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: Bootstrap starting.
I20260620 15:00:01.152674 3956 tablet_bootstrap.cc:492] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: Bootstrap starting.
I20260620 15:00:01.153959 3956 tablet_bootstrap.cc:654] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:01.154287 3957 tablet_bootstrap.cc:654] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:01.156324 3955 tablet_bootstrap.cc:492] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: Bootstrap starting.
I20260620 15:00:01.157979 3956 tablet_bootstrap.cc:492] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: No bootstrap required, opened a new log
I20260620 15:00:01.158128 3956 ts_tablet_manager.cc:1397] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260620 15:00:01.158143 3980 tablet_replica.cc:330] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: stopping tablet replica
I20260620 15:00:01.158707 3955 tablet_bootstrap.cc:654] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:01.158965 3956 raft_consensus.cc:348] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.159183 3956 raft_consensus.cc:374] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:01.159281 3956 raft_consensus.cc:729] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a3858f936d495b8d816a76b20627eb, State: Initialized, Role: FOLLOWER
I20260620 15:00:01.159552 3956 consensus_queue.cc:260] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.160040 3980 raft_consensus.cc:2229] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:01.161046 3980 raft_consensus.cc:2258] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:01.163755 3980 ts_tablet_manager.cc:1905] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:01.163873 3956 ts_tablet_manager.cc:1428] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260620 15:00:01.164069 3979 ts_tablet_manager.cc:1918] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:01.164264 3979 log.cc:1196] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/c17261a4644d4b18847a2dc68b63fa3f
I20260620 15:00:01.167906 3979 ts_tablet_manager.cc:1939] T c17261a4644d4b18847a2dc68b63fa3f P 12a3858f936d495b8d816a76b20627eb: Deleting consensus metadata
I20260620 15:00:01.168459 3957 tablet_bootstrap.cc:492] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: No bootstrap required, opened a new log
I20260620 15:00:01.168604 3957 ts_tablet_manager.cc:1397] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260620 15:00:01.168839 3955 tablet_bootstrap.cc:492] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: No bootstrap required, opened a new log
I20260620 15:00:01.168992 3955 ts_tablet_manager.cc:1397] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260620 15:00:01.169531 3957 raft_consensus.cc:348] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.169771 3957 raft_consensus.cc:374] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:01.169750 3437 catalog_manager.cc:4841] TS 12a3858f936d495b8d816a76b20627eb (127.2.70.1:45183): tablet c17261a4644d4b18847a2dc68b63fa3f (table test-table [id=f81f7aaf3c4a43e1a36d3b79f95964d2]) successfully deleted
I20260620 15:00:01.169869 3957 raft_consensus.cc:729] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcc2f08341043d0b96242b7deeeb888, State: Initialized, Role: FOLLOWER
I20260620 15:00:01.169752 3955 raft_consensus.cc:348] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.169982 3955 raft_consensus.cc:374] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:01.170075 3955 raft_consensus.cc:729] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc4c0ab4a2843769b004cec92ae84c0, State: Initialized, Role: FOLLOWER
I20260620 15:00:01.170159 3957 consensus_queue.cc:260] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.170724 3957 ts_tablet_manager.cc:1428] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 15:00:01.171031 3978 ts_tablet_manager.cc:1918] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:01.171193 3978 log.cc:1196] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/c17261a4644d4b18847a2dc68b63fa3f
I20260620 15:00:01.171751 3978 ts_tablet_manager.cc:1939] T c17261a4644d4b18847a2dc68b63fa3f P cbc4c0ab4a2843769b004cec92ae84c0: Deleting consensus metadata
I20260620 15:00:01.173547 3438 catalog_manager.cc:4841] TS cbc4c0ab4a2843769b004cec92ae84c0 (127.2.70.3:43373): tablet c17261a4644d4b18847a2dc68b63fa3f (table test-table [id=f81f7aaf3c4a43e1a36d3b79f95964d2]) successfully deleted
I20260620 15:00:01.170377 3955 consensus_queue.cc:260] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.175529 3980 ts_tablet_manager.cc:1918] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:01.175673 3980 log.cc:1196] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/c17261a4644d4b18847a2dc68b63fa3f
I20260620 15:00:01.176195 3980 ts_tablet_manager.cc:1939] T c17261a4644d4b18847a2dc68b63fa3f P bdcc2f08341043d0b96242b7deeeb888: Deleting consensus metadata
I20260620 15:00:01.176286 3955 ts_tablet_manager.cc:1428] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260620 15:00:01.177910 3437 catalog_manager.cc:4841] TS bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057): tablet c17261a4644d4b18847a2dc68b63fa3f (table test-table [id=f81f7aaf3c4a43e1a36d3b79f95964d2]) successfully deleted
I20260620 15:00:01.574812 3962 raft_consensus.cc:482] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:01.575052 3962 raft_consensus.cc:504] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.576586 3962 leader_election.cc:288] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12a3858f936d495b8d816a76b20627eb (127.2.70.1:45183), bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057)
I20260620 15:00:01.587327 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.577049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:58225 (local address 127.2.70.1:45183)
0620 15:00:01.577249 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.577257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.577652 (+ 395us) server_negotiation.cc:407] Connection header received
0620 15:00:01.577973 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.577983 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.578078 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.578164 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:01.579026 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.580578 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.583901 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.585673 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.585826 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.585994 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.586666 (+ 672us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.586883 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:00:01.588647 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.576964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:45183 (local address 127.2.70.3:58225)
0620 15:00:01.577517 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:01.577535 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:01.577777 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:01.578417 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:01.578424 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:01.578478 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:01.578775 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.578791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.580837 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.580845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.583650 (+ 2805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.583671 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.586164 (+ 2493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.586173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.586433 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.586486 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 15:00:01.587900 (+ 1414us) client_negotiation.cc:240] Negotiation successful
0620 15:00:01.588121 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260620 15:00:01.589970 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c413fd84d74e50a9c29f7c671e551a" candidate_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a3858f936d495b8d816a76b20627eb" is_pre_election: true
I20260620 15:00:01.590409 3613 raft_consensus.cc:2454] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbc4c0ab4a2843769b004cec92ae84c0 in term 0.
I20260620 15:00:01.591091 3819 leader_election.cc:302] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3858f936d495b8d816a76b20627eb, cbc4c0ab4a2843769b004cec92ae84c0; no voters:
I20260620 15:00:01.591526 3962 raft_consensus.cc:2781] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:01.591650 3962 raft_consensus.cc:482] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:01.591755 3962 raft_consensus.cc:3037] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.595096 3984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.577709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:36057 (local address 127.2.70.3:35169)
0620 15:00:01.589125 (+ 11416us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:01.589144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:01.589283 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:01.590073 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:01.590077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:01.590103 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:01.590269 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.590279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.591537 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.591543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.593103 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.593120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.594326 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.594331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.594523 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.594546 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:00:01.594650 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:00:01.594783 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11303,"thread_start_us":11228,"threads_started":1}
I20260620 15:00:01.595106 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.577826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:35169 (local address 127.2.70.2:36057)
0620 15:00:01.589765 (+ 11939us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.589770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.589785 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:00:01.589839 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.589843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.589934 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.590056 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:01.590425 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.591417 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.593292 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.594168 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.594303 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.594394 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.594674 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.594801 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11816,"thread_start_us":11727,"threads_started":1}
I20260620 15:00:01.595307 3962 raft_consensus.cc:504] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.596019 3962 leader_election.cc:288] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [CANDIDATE]: Term 1 election: Requested vote from peers 12a3858f936d495b8d816a76b20627eb (127.2.70.1:45183), bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057)
I20260620 15:00:01.595983 3746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c413fd84d74e50a9c29f7c671e551a" candidate_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcc2f08341043d0b96242b7deeeb888" is_pre_election: true
I20260620 15:00:01.596278 3746 raft_consensus.cc:2454] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbc4c0ab4a2843769b004cec92ae84c0 in term 0.
I20260620 15:00:01.596454 3613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c413fd84d74e50a9c29f7c671e551a" candidate_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a3858f936d495b8d816a76b20627eb"
I20260620 15:00:01.596678 3613 raft_consensus.cc:3037] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.597164 3746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c413fd84d74e50a9c29f7c671e551a" candidate_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcc2f08341043d0b96242b7deeeb888"
I20260620 15:00:01.597335 3746 raft_consensus.cc:3037] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.599467 3746 raft_consensus.cc:2454] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc4c0ab4a2843769b004cec92ae84c0 in term 1.
I20260620 15:00:01.599985 3819 leader_election.cc:302] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdcc2f08341043d0b96242b7deeeb888, cbc4c0ab4a2843769b004cec92ae84c0; no voters:
I20260620 15:00:01.600060 3613 raft_consensus.cc:2454] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbc4c0ab4a2843769b004cec92ae84c0 in term 1.
I20260620 15:00:01.600309 3962 raft_consensus.cc:2781] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:01.600802 3962 raft_consensus.cc:686] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 LEADER]: Becoming Leader. State: Replica: cbc4c0ab4a2843769b004cec92ae84c0, State: Running, Role: LEADER
I20260620 15:00:01.601230 3962 consensus_queue.cc:237] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.604815 3451 catalog_manager.cc:5495] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cbc4c0ab4a2843769b004cec92ae84c0 (127.2.70.3). New cstate: current_term: 1 leader_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } health_report { overall_health: HEALTHY } } }
Jun 20 15:00:01 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967598, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:01.650981 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.635916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42221 (local address 127.0.0.1:36692)
0620 15:00:01.636775 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:01.636803 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:01.636999 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:01.637438 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:01.637443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:01.638185 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:01.638409 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.638419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.639553 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.639560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.640586 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.640597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.641451 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.641457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.641681 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.642357 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:01.642378 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:01.644644 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:01.647843 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:01.647852 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:01.647857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:01.648681 (+ 824us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:01.649031 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:01.649035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:01.649037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:01.649170 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:01.649803 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:01.649820 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:01.649982 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 15:00:01.650157 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 15:00:01.650465 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":458,"threads_started":1}
I20260620 15:00:01.650983 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.635955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.2.70.62:42221)
0620 15:00:01.636765 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.636772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.636911 (+ 139us) server_negotiation.cc:407] Connection header received
0620 15:00:01.637135 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.637141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.637209 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.637329 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:01.638600 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.639410 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.640814 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.641267 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.641434 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.644880 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:01.644885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:01.644889 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:01.644918 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:01.647724 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:01.648798 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:01.648803 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:01.648805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:01.648933 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:01.649259 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:01.649263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:01.649265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:01.649682 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:01.649831 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.650246 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.650601 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":635,"thread_start_us":425,"threads_started":1}
I20260620 15:00:01.656221 3451 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:36692:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:00:01.657389 3451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:01.664278 3866 tablet_service.cc:1467] Processing CreateTablet for tablet c65a0b050778497b982ae02bf88e1a86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f403f24a1ca4802babf20b7a5d2152e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:01.664278 3727 tablet_service.cc:1467] Processing CreateTablet for tablet c65a0b050778497b982ae02bf88e1a86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f403f24a1ca4802babf20b7a5d2152e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:01.664281 3588 tablet_service.cc:1467] Processing CreateTablet for tablet c65a0b050778497b982ae02bf88e1a86 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6f403f24a1ca4802babf20b7a5d2152e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:01.664853 3588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65a0b050778497b982ae02bf88e1a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:01.664881 3727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65a0b050778497b982ae02bf88e1a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:01.664907 3866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65a0b050778497b982ae02bf88e1a86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:01.671247 3996 tablet_bootstrap.cc:492] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: Bootstrap starting.
I20260620 15:00:01.672778 3996 tablet_bootstrap.cc:654] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:01.673338 3955 tablet_bootstrap.cc:492] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: Bootstrap starting.
I20260620 15:00:01.675773 3955 tablet_bootstrap.cc:654] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:01.677685 3996 tablet_bootstrap.cc:492] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: No bootstrap required, opened a new log
I20260620 15:00:01.678074 3996 ts_tablet_manager.cc:1397] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260620 15:00:01.679052 3997 tablet_bootstrap.cc:492] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: Bootstrap starting.
I20260620 15:00:01.678927 3996 raft_consensus.cc:348] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.679154 3996 raft_consensus.cc:374] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:01.679242 3996 raft_consensus.cc:729] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a3858f936d495b8d816a76b20627eb, State: Initialized, Role: FOLLOWER
I20260620 15:00:01.679469 3996 consensus_queue.cc:260] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.680331 3996 ts_tablet_manager.cc:1428] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:01.680871 3997 tablet_bootstrap.cc:654] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:01.681202 4000 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.681875 4000 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:01.689929 3997 tablet_bootstrap.cc:492] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: No bootstrap required, opened a new log
I20260620 15:00:01.690093 3997 ts_tablet_manager.cc:1397] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260620 15:00:01.690220 3955 tablet_bootstrap.cc:492] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: No bootstrap required, opened a new log
I20260620 15:00:01.690389 3955 ts_tablet_manager.cc:1397] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260620 15:00:01.690862 3997 raft_consensus.cc:348] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.691103 3997 raft_consensus.cc:374] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:01.691210 3997 raft_consensus.cc:729] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdcc2f08341043d0b96242b7deeeb888, State: Initialized, Role: FOLLOWER
I20260620 15:00:01.691134 3955 raft_consensus.cc:348] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.691361 3955 raft_consensus.cc:374] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:01.691447 3955 raft_consensus.cc:729] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc4c0ab4a2843769b004cec92ae84c0, State: Initialized, Role: FOLLOWER
I20260620 15:00:01.691471 3997 consensus_queue.cc:260] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.691661 3955 consensus_queue.cc:260] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.692042 3962 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.692293 3962 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:01.692711 3955 ts_tablet_manager.cc:1428] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 15:00:01.693825 3997 ts_tablet_manager.cc:1428] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:01.693291 4002 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.694079 4002 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:01.737107 4000 raft_consensus.cc:482] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:01.737325 4000 raft_consensus.cc:504] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.738140 4000 leader_election.cc:288] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057), cbc4c0ab4a2843769b004cec92ae84c0 (127.2.70.3:43373)
I20260620 15:00:01.738699 3746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65a0b050778497b982ae02bf88e1a86" candidate_uuid: "12a3858f936d495b8d816a76b20627eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcc2f08341043d0b96242b7deeeb888" is_pre_election: true
I20260620 15:00:01.738732 3889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65a0b050778497b982ae02bf88e1a86" candidate_uuid: "12a3858f936d495b8d816a76b20627eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" is_pre_election: true
I20260620 15:00:01.738993 3746 raft_consensus.cc:2454] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a3858f936d495b8d816a76b20627eb in term 0.
I20260620 15:00:01.739027 3889 raft_consensus.cc:2454] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a3858f936d495b8d816a76b20627eb in term 0.
I20260620 15:00:01.739490 3541 leader_election.cc:302] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3858f936d495b8d816a76b20627eb, cbc4c0ab4a2843769b004cec92ae84c0; no voters:
I20260620 15:00:01.739826 4000 raft_consensus.cc:2781] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:01.739960 4000 raft_consensus.cc:482] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:01.740099 4000 raft_consensus.cc:3037] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.742254 4000 raft_consensus.cc:504] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.742895 4000 leader_election.cc:288] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [CANDIDATE]: Term 1 election: Requested vote from peers bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057), cbc4c0ab4a2843769b004cec92ae84c0 (127.2.70.3:43373)
I20260620 15:00:01.743348 3746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65a0b050778497b982ae02bf88e1a86" candidate_uuid: "12a3858f936d495b8d816a76b20627eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdcc2f08341043d0b96242b7deeeb888"
I20260620 15:00:01.743410 3889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65a0b050778497b982ae02bf88e1a86" candidate_uuid: "12a3858f936d495b8d816a76b20627eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbc4c0ab4a2843769b004cec92ae84c0"
I20260620 15:00:01.743615 3889 raft_consensus.cc:3037] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.743615 3746 raft_consensus.cc:3037] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:01.746150 3746 raft_consensus.cc:2454] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a3858f936d495b8d816a76b20627eb in term 1.
I20260620 15:00:01.746709 3540 leader_election.cc:302] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a3858f936d495b8d816a76b20627eb, bdcc2f08341043d0b96242b7deeeb888; no voters:
I20260620 15:00:01.746862 3889 raft_consensus.cc:2454] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a3858f936d495b8d816a76b20627eb in term 1.
I20260620 15:00:01.747107 4000 raft_consensus.cc:2781] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:01.747637 4000 raft_consensus.cc:686] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [term 1 LEADER]: Becoming Leader. State: Replica: 12a3858f936d495b8d816a76b20627eb, State: Running, Role: LEADER
I20260620 15:00:01.747997 4000 consensus_queue.cc:237] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } }
I20260620 15:00:01.749652 4003 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: TxnStatusTablet state changed. Reason: New leader 12a3858f936d495b8d816a76b20627eb. Latest consensus state: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.749948 4003 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:01.750685 4005 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:00:01.750937 4005 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:00:01.751230 3449 catalog_manager.cc:5495] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 12a3858f936d495b8d816a76b20627eb (127.2.70.1). New cstate: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:01.777354 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.771792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.2.70.1:45183)
0620 15:00:01.771950 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.771957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.772045 (+ 88us) server_negotiation.cc:407] Connection header received
0620 15:00:01.772302 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.772306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.772394 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.772467 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:01.772877 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.773760 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.775652 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.776126 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.776216 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.776738 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:01.776839 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:01.776909 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.777076 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.777137 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 15:00:01.777618 3995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.771696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:45183 (local address 127.0.0.1:48212)
0620 15:00:01.771949 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:01.771970 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:01.772125 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:01.772524 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:01.772529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:01.772544 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:01.772737 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.772749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.773902 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.773908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.775524 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.775540 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.776280 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.776289 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.776501 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.776578 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:01.776941 (+ 363us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:01.776963 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:00:01.777053 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:00:01.777213 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:00:01.781822 3746 raft_consensus.cc:1261] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Refusing update from remote peer 12a3858f936d495b8d816a76b20627eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:01.781816 3889 raft_consensus.cc:1261] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Refusing update from remote peer 12a3858f936d495b8d816a76b20627eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:01.782455 4003 consensus_queue.cc:1035] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:01.782966 4000 consensus_queue.cc:1035] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:01.789110 4008 mvcc.cc:205] Tried to move back new op lower bound from 7298939296891846656 to 7298939296762060800. Current Snapshot: MvccSnapshot[applied={T|T < 7298939296891846656}]
I20260620 15:00:01.792716 4003 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.793118 4003 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:01.800544 3962 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: TxnStatusTablet state changed. Reason: New leader 12a3858f936d495b8d816a76b20627eb. Latest consensus state: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.800838 3962 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:01.805047 4000 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.805424 4000 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P 12a3858f936d495b8d816a76b20627eb: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:01.808954 4002 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: TxnStatusTablet state changed. Reason: New leader 12a3858f936d495b8d816a76b20627eb. Latest consensus state: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.809288 4002 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:01.810122 4009 mvcc.cc:205] Tried to move back new op lower bound from 7298939296891846656 to 7298939296762060800. Current Snapshot: MvccSnapshot[applied={T|T < 7298939296891846656}]
I20260620 15:00:01.811149 4014 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.811517 4014 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P bdcc2f08341043d0b96242b7deeeb888: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:01.811932 3962 tablet_replica.cc:439] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "12a3858f936d495b8d816a76b20627eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 } } peers { permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 } } peers { permanent_uuid: "cbc4c0ab4a2843769b004cec92ae84c0" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 43373 } } }
I20260620 15:00:01.812265 3962 tablet_replica.cc:442] T c65a0b050778497b982ae02bf88e1a86 P cbc4c0ab4a2843769b004cec92ae84c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:01.826030 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.818693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.2.70.3:43373)
0620 15:00:01.819729 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.819737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.820225 (+ 488us) server_negotiation.cc:407] Connection header received
0620 15:00:01.820447 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.820451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.820543 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.820720 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:01.821345 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.822163 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.823608 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.824053 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.824201 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.824650 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:01.825318 (+ 668us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:01.825492 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.825575 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.825666 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":344,"threads_started":1}
I20260620 15:00:01.842942 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.836802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:45183 (local address 127.2.70.3:47905)
0620 15:00:01.837397 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:01.837413 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:01.837594 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:01.837929 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:01.837932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:01.837941 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:01.838184 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.838191 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.839643 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.839647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.840713 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.840724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.841810 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.841815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.841942 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.841994 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:01.842467 (+ 473us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:01.842483 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 15:00:01.842551 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 15:00:01.842652 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":294,"threads_started":1}
I20260620 15:00:01.843003 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.836941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:47905 (local address 127.2.70.1:45183)
0620 15:00:01.837124 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.837131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.837522 (+ 391us) server_negotiation.cc:407] Connection header received
0620 15:00:01.837722 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.837728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.837813 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.837915 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:01.838321 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.839508 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.840849 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.841625 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.841743 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.842164 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:01.842350 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:01.842451 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.842599 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.842679 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:00:01.852669 3747 raft_consensus.cc:1261] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Refusing update from remote peer cbc4c0ab4a2843769b004cec92ae84c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:01.853050 3613 raft_consensus.cc:1261] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Refusing update from remote peer cbc4c0ab4a2843769b004cec92ae84c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:01.853269 3987 consensus_queue.cc:1035] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdcc2f08341043d0b96242b7deeeb888" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:01.853724 3962 consensus_queue.cc:1035] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a3858f936d495b8d816a76b20627eb" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:01.892321 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.884657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:37381 (local address 127.2.70.3:43373)
0620 15:00:01.884935 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.884942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.885773 (+ 831us) server_negotiation.cc:407] Connection header received
0620 15:00:01.886037 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.886042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.886112 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.886205 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:01.886776 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.887918 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.890044 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.890582 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.890728 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.891149 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:01.891297 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:01.891402 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.891531 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.891608 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260620 15:00:01.892549 4027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.884768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:43373 (local address 127.2.70.1:37381)
0620 15:00:01.885677 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:01.885697 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:01.885897 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:01.886218 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:01.886223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:01.886233 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:01.886631 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.886644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.888067 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.888072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.889920 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:01.889934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.890742 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.890748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:01.890912 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.890967 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:01.891404 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:01.891425 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:00:01.891504 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 15:00:01.891646 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":442,"threads_started":1}
I20260620 15:00:01.931622 3451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:01.931882 3451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:01.935312 3451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f93932b6b8a4e4885eedf3d3989120a: Sending DeleteTablet for 3 replicas of tablet 61c413fd84d74e50a9c29f7c671e551a
I20260620 15:00:01.936332 3866 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c413fd84d74e50a9c29f7c671e551a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:01 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:49848
I20260620 15:00:01.936422 3727 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c413fd84d74e50a9c29f7c671e551a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:01 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:40526
I20260620 15:00:01.938215 3588 tablet_service.cc:1514] Processing DeleteTablet for tablet 61c413fd84d74e50a9c29f7c671e551a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:01 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:55516
I20260620 15:00:01.949505 4031 tablet_replica.cc:330] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: stopping tablet replica
I20260620 15:00:01.949743 4031 raft_consensus.cc:2229] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:01.950377 4031 raft_consensus.cc:2258] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:01.951623 4031 ts_tablet_manager.cc:1905] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:01.962857 4031 ts_tablet_manager.cc:1918] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:01.963021 4031 log.cc:1196] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/61c413fd84d74e50a9c29f7c671e551a
I20260620 15:00:01.963553 4031 ts_tablet_manager.cc:1939] T 61c413fd84d74e50a9c29f7c671e551a P cbc4c0ab4a2843769b004cec92ae84c0: Deleting consensus metadata
I20260620 15:00:01.964771 3438 catalog_manager.cc:4841] TS cbc4c0ab4a2843769b004cec92ae84c0 (127.2.70.3:43373): tablet 61c413fd84d74e50a9c29f7c671e551a (table test-table [id=f0c03b1a16a746c38cede509e80b91b4]) successfully deleted
I20260620 15:00:01.965960 4034 tablet_replica.cc:330] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: stopping tablet replica
I20260620 15:00:01.966277 4034 raft_consensus.cc:2229] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:01.966745 4034 raft_consensus.cc:2258] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:01.967993 4032 tablet_replica.cc:330] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: stopping tablet replica
I20260620 15:00:01.968564 4034 ts_tablet_manager.cc:1905] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:01.968979 4032 raft_consensus.cc:2229] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:01.970270 4032 raft_consensus.cc:2258] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:01.972036 4032 ts_tablet_manager.cc:1905] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:00:01 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:00:01.979787 4032 ts_tablet_manager.cc:1918] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:01.979918 4032 log.cc:1196] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/61c413fd84d74e50a9c29f7c671e551a
I20260620 15:00:01.980376 4032 ts_tablet_manager.cc:1939] T 61c413fd84d74e50a9c29f7c671e551a P bdcc2f08341043d0b96242b7deeeb888: Deleting consensus metadata
I20260620 15:00:01.981624 3437 catalog_manager.cc:4841] TS bdcc2f08341043d0b96242b7deeeb888 (127.2.70.2:36057): tablet 61c413fd84d74e50a9c29f7c671e551a (table test-table [id=f0c03b1a16a746c38cede509e80b91b4]) successfully deleted
I20260620 15:00:01.984036 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.946791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.2.70.1:45183)
0620 15:00:01.947045 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.947054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.962893 (+ 15839us) server_negotiation.cc:407] Connection header received
0620 15:00:01.963534 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.963541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.963630 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.963712 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:01.965809 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.967030 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.969001 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.970537 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.970663 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.975775 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:01.975792 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:01.975812 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:01.975857 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:01.980713 (+ 4856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:01.981686 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:01.981695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:01.981698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:01.981941 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:01.982351 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:01.982357 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:01.982369 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:01.982740 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:01.983101 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:01.983397 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 15:00:01.983636 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:00:01.984941 4034 ts_tablet_manager.cc:1918] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
W20260620 15:00:01.984977 3653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48226
I20260620 15:00:01.985091 4034 log.cc:1196] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/61c413fd84d74e50a9c29f7c671e551a
I20260620 15:00:01.985623 4034 ts_tablet_manager.cc:1939] T 61c413fd84d74e50a9c29f7c671e551a P 12a3858f936d495b8d816a76b20627eb: Deleting consensus metadata
I20260620 15:00:01.986692 3437 catalog_manager.cc:4841] TS 12a3858f936d495b8d816a76b20627eb (127.2.70.1:45183): tablet 61c413fd84d74e50a9c29f7c671e551a (table test-table [id=f0c03b1a16a746c38cede509e80b91b4]) successfully deleted
I20260620 15:00:02.006045 3994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:01.988381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.2.70.62:42221)
0620 15:00:01.988681 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:01.988686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:01.989246 (+ 560us) server_negotiation.cc:407] Connection header received
0620 15:00:01.989412 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:01.989418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:01.989526 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:01.989630 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:01.991387 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.992676 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.994490 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:01.995189 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:01.995329 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:01.998331 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:01.998340 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:01.998347 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:01.998390 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:02.002703 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:02.003654 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:02.003660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:02.003664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:02.003821 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:02.004312 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:02.004318 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:02.004322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:02.004715 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:02.004989 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:02.005367 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 15:00:02.005676 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260620 15:00:02.006652 3451 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36706
I20260620 15:00:02.009301 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3522
I20260620 15:00:02.031085 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:02.011336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48230 (local address 127.2.70.1:45183)
0620 15:00:02.011525 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:02.011531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:02.014008 (+ 2477us) server_negotiation.cc:407] Connection header received
0620 15:00:02.014172 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:02.014178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:02.014262 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:02.014389 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:02.015867 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:02.017107 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:02.019072 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:02.019798 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:02.019916 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:02.023242 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:02.023249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:02.023253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:02.023300 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:02.027754 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:02.028602 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:02.028608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:02.028612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:02.028781 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:02.029260 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:02.029267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:02.029275 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:02.029644 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:02.029892 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:02.030412 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 15:00:02.030654 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 15:00:02.032440 3653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48230
W20260620 15:00:02.032999 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3522: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:02.033185 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 3522
I20260620 15:00:02.057549 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3662
Jun 20 15:00:02 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:00:02.074200 3986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:02.058352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.2.70.2:36057)
0620 15:00:02.058529 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:02.058533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:02.058551 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:00:02.058726 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:02.058730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:02.058801 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:02.058923 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:02.060371 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:02.061067 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:02.062931 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:02.063311 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:02.063425 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:02.068234 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:02.068249 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:02.068265 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:02.068298 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:02.071515 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:02.072194 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:02.072202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:02.072204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:02.072328 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:02.072697 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:02.072702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:02.072705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:02.073055 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:02.073349 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:02.073649 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 15:00:02.073877 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 15:00:02.074818 3792 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45676
W20260620 15:00:02.075457 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3662: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:02.075659 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 3662
I20260620 15:00:02.093087 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3801
Jun 20 15:00:02 dist-test-slave-n5vv krb5kdc[3406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967600, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:00:02.109344 4018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:02.094034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.2.70.3:43373)
0620 15:00:02.094214 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:02.094219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:02.094236 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:00:02.094291 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:02.094295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:02.094354 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:02.094441 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:02.096097 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:02.096826 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:02.098663 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:02.099102 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:02.099192 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:02.103587 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:02.103600 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:02.103617 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:02.103647 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:02.106705 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:02.107352 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:02.107360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:02.107362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:02.107488 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:02.107851 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:02.107856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:02.107859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:02.108226 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:02.108564 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:02.108791 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 15:00:02.109028 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 15:00:02.110028 3931 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57356
W20260620 15:00:02.110703 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3801: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:02.110854 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 3801
I20260620 15:00:02.128877 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 3419
I20260620 15:00:02.130097 3491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:02.243332 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 3419
2026-06-20T15:00:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4933 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:02 dist-test-slave-n5vv krb5kdc[4042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:02 dist-test-slave-n5vv krb5kdc[4042](info): set up 2 sockets
Jun 20 15:00:02 dist-test-slave-n5vv krb5kdc[4042](info): commencing operation
krb5kdc: starting...
W20260620 15:00:03.325778 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:03 dist-test-slave-n5vv krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:03.521392 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42913
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:42913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:03.844074 4055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:03.844359 4055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:03.844519 4055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:03.851905 4055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:03.852012 4055 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:00:03.852104 4055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:03.852216 4055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:03.852306 4055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:03.852397 4055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:03.862365 4055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:42913
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:03.863687 4055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:03.865021 4055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:03.876987 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:03.876938 4061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:03.877053 4060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:03.878201 4055 server_base.cc:1034] running on GCE node
I20260620 15:00:03.878955 4055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:03.880239 4055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:03.881453 4055 hybrid_clock.cc:648] HybridClock initialized: now 1781967603881399 us; error 33 us; skew 500 ppm
I20260620 15:00:03.881791 4055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:03 dist-test-slave-n5vv krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:03.884738 4055 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:00:03.886276 4055 webserver.cc:460] Webserver started at http://127.2.70.62:43959/ using document root <none> and password file <none>
I20260620 15:00:03.887007 4055 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:03.887104 4055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:03.887404 4055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:03.890409 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "789308503fe24b73b4d37ff3ddeb6298"
format_stamp: "Formatted at 2026-06-20 15:00:03 on dist-test-slave-n5vv"
server_key: "3ad72cb9f7e20d5d7062f848af018370"
server_key_iv: "6f3c8bf2ad04831debb3c3150a205251"
server_key_version: "encryptionkey@0"
I20260620 15:00:03.891129 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "789308503fe24b73b4d37ff3ddeb6298"
format_stamp: "Formatted at 2026-06-20 15:00:03 on dist-test-slave-n5vv"
server_key: "3ad72cb9f7e20d5d7062f848af018370"
server_key_iv: "6f3c8bf2ad04831debb3c3150a205251"
server_key_version: "encryptionkey@0"
I20260620 15:00:03.896170 4055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:00:03.899624 4070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:03.901325 4055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:00:03.901553 4055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "789308503fe24b73b4d37ff3ddeb6298"
format_stamp: "Formatted at 2026-06-20 15:00:03 on dist-test-slave-n5vv"
server_key: "3ad72cb9f7e20d5d7062f848af018370"
server_key_iv: "6f3c8bf2ad04831debb3c3150a205251"
server_key_version: "encryptionkey@0"
I20260620 15:00:03.901769 4055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:03.922857 4055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:03.927044 4055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:03.927340 4055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:03.948997 4055 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:42913
I20260620 15:00:03.949016 4131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:42913 every 8 connection(s)
I20260620 15:00:03.950491 4055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:03.954334 4132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:03.960100 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 4055
I20260620 15:00:03.960310 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:03.961140 2328 external_mini_cluster.cc:1428] Setting key 10fd0693ddc827775a48d262852ba95a
I20260620 15:00:03.965909 4132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: Bootstrap starting.
I20260620 15:00:03.969652 4132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:03.970845 4132 log.cc:824] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:03.974278 4132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: No bootstrap required, opened a new log
Jun 20 15:00:03 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:03.982772 4132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789308503fe24b73b4d37ff3ddeb6298" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42913 } }
I20260620 15:00:03.983187 4132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:03.983320 4132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789308503fe24b73b4d37ff3ddeb6298, State: Initialized, Role: FOLLOWER
I20260620 15:00:03.984033 4132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789308503fe24b73b4d37ff3ddeb6298" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42913 } }
I20260620 15:00:03.984292 4132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:03.984437 4132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:03.984638 4132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:03.985296 4135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:03.963428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38366 (local address 127.2.70.62:42913)
0620 15:00:03.964272 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:03.964284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:03.964325 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:00:03.966133 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:03.966155 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:03.966550 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:03.966874 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:03.968713 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:03.970160 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:03.971980 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:03.972778 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:03.972922 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:03.977666 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:03.977674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:03.977689 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:03.977732 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:03.980923 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:03.981724 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:03.981732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:03.981738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:03.981905 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:03.982275 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:03.982281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:03.982284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:03.982825 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:03.983135 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:03.983578 (+ 443us) server_negotiation.cc:299] Negotiation successful
0620 15:00:03.983989 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":420,"threads_started":1}
I20260620 15:00:03.987568 4132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789308503fe24b73b4d37ff3ddeb6298" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42913 } }
I20260620 15:00:03.988112 4132 leader_election.cc:302] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 789308503fe24b73b4d37ff3ddeb6298; no voters:
I20260620 15:00:03.988718 4132 leader_election.cc:288] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:03.988860 4137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:03.989594 4137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [term 1 LEADER]: Becoming Leader. State: Replica: 789308503fe24b73b4d37ff3ddeb6298, State: Running, Role: LEADER
I20260620 15:00:03.990154 4137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789308503fe24b73b4d37ff3ddeb6298" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42913 } }
I20260620 15:00:03.990779 4132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:03.992733 4138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "789308503fe24b73b4d37ff3ddeb6298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789308503fe24b73b4d37ff3ddeb6298" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42913 } } }
I20260620 15:00:03.993105 4138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:03.993323 4139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789308503fe24b73b4d37ff3ddeb6298. Latest consensus state: current_term: 1 leader_uuid: "789308503fe24b73b4d37ff3ddeb6298" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789308503fe24b73b4d37ff3ddeb6298" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42913 } } }
I20260620 15:00:03.993754 4139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:03.995180 4142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:04.002079 4142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:04.011941 4142 catalog_manager.cc:1347] Generated new cluster ID: 8b8199783c7e4170b30d91454bf92cab
I20260620 15:00:04.012079 4142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:04.021278 4142 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:04.023312 4142 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:04.033883 4142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: Generated new TSK 0
I20260620 15:00:04.034767 4142 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:00:04.086129 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42913
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:04.412997 4158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:04.413273 4158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:04.413372 4158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:04.420583 4158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:04.420694 4158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:04.420778 4158 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:00:04.420828 4158 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:00:04.420888 4158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:04.430655 4158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:04.431914 4158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:04.433140 4158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:04.445996 4163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:04.446067 4164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:04.446038 4166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:04.447561 4158 server_base.cc:1034] running on GCE node
I20260620 15:00:04.448186 4158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:04.449120 4158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:04.450348 4158 hybrid_clock.cc:648] HybridClock initialized: now 1781967604450295 us; error 47 us; skew 500 ppm
I20260620 15:00:04.450704 4158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:04 dist-test-slave-n5vv krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:04.453785 4158 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:00:04.455179 4158 webserver.cc:460] Webserver started at http://127.2.70.1:38303/ using document root <none> and password file <none>
I20260620 15:00:04.455814 4158 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:04.455911 4158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:04.456152 4158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:04.458928 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "8ed26b54b84b4c33af528ce20bfc0514"
format_stamp: "Formatted at 2026-06-20 15:00:04 on dist-test-slave-n5vv"
server_key: "fb4a796f6f3f4faea68fd3e56739d553"
server_key_iv: "ecfc27ffdaa4c1c0d13aaef7c5a91048"
server_key_version: "encryptionkey@0"
I20260620 15:00:04.459611 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "8ed26b54b84b4c33af528ce20bfc0514"
format_stamp: "Formatted at 2026-06-20 15:00:04 on dist-test-slave-n5vv"
server_key: "fb4a796f6f3f4faea68fd3e56739d553"
server_key_iv: "ecfc27ffdaa4c1c0d13aaef7c5a91048"
server_key_version: "encryptionkey@0"
I20260620 15:00:04.464370 4158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 15:00:04.467725 4173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:04.468987 4158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:00:04.469152 4158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "8ed26b54b84b4c33af528ce20bfc0514"
format_stamp: "Formatted at 2026-06-20 15:00:04 on dist-test-slave-n5vv"
server_key: "fb4a796f6f3f4faea68fd3e56739d553"
server_key_iv: "ecfc27ffdaa4c1c0d13aaef7c5a91048"
server_key_version: "encryptionkey@0"
I20260620 15:00:04.469324 4158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:04.492899 4158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:04.497313 4158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:04.497633 4158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:04.500245 4158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:04.500387 4158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:04.500520 4158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:04.500615 4158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:00:04 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:04.531076 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:04.500965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:36937 (local address 127.2.70.62:42913)
0620 15:00:04.503958 (+ 2993us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:04.503962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:04.503980 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:00:04.507038 (+ 3058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:04.507045 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:04.507217 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:04.507367 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:04.511126 (+ 3759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:04.512362 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:04.515569 (+ 3207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:04.516418 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:04.516588 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:04.522411 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:04.522421 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:04.522428 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:04.522485 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:04.526707 (+ 4222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:04.528015 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:04.528022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:04.528025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:04.528186 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:04.528790 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:04.528795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:04.528799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:04.529184 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:04.529354 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:04.530585 (+ 1231us) server_negotiation.cc:299] Negotiation successful
0620 15:00:04.530811 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2883,"thread_start_us":2781,"threads_started":1}
I20260620 15:00:04.533038 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:04.501833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42913 (local address 127.2.70.1:36937)
0620 15:00:04.503324 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:04.503389 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:04.506664 (+ 3275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:04.508212 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:04.508233 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:04.509809 (+ 1576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:04.510871 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:04.510897 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:04.512506 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:04.512512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:04.515387 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:04.515409 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:04.516702 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:04.516715 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:04.517097 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:04.518065 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:04.518115 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:04.522061 (+ 3946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:04.526909 (+ 4848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:04.526923 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:04.526953 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:04.527846 (+ 893us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:04.528396 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:04.528403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:04.528409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:04.528658 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:04.529354 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:04.529365 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:04.530021 (+ 656us) client_negotiation.cc:769] Sending connection context
0620 15:00:04.530458 (+ 437us) client_negotiation.cc:240] Negotiation successful
0620 15:00:04.531074 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":560,"threads_started":1}
I20260620 15:00:04.539821 4292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:43091 every 8 connection(s)
I20260620 15:00:04.539819 4158 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:43091
I20260620 15:00:04.541450 4158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:04.543470 4294 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42913
I20260620 15:00:04.543702 4294 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:04.544310 4294 heartbeater.cc:511] Master 127.2.70.62:42913 requested a full tablet report, sending...
I20260620 15:00:04.546057 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 4158
I20260620 15:00:04.546248 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:04.546299 4087 ts_manager.cc:194] Registered new tserver with Master: 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1:43091)
I20260620 15:00:04.547026 2328 external_mini_cluster.cc:1428] Setting key d1605345451565848ca5f9cf4d13ff79
I20260620 15:00:04.547911 4087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:36937
I20260620 15:00:04.554821 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:04.541025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42913 (local address 127.2.70.1:44387)
0620 15:00:04.541744 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:04.541763 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:04.541904 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:04.542481 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:04.542486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:04.543170 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:04.543530 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:04.543560 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:04.544559 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:04.544564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:04.545889 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:04.545904 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:04.547148 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:04.547156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:04.547321 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:04.547959 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:04.547982 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:04.548747 (+ 765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:04.552594 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:04.552599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:04.552601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:04.552955 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:04.553345 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:04.553349 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:04.553351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:04.553457 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:04.554087 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:04.554090 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:04.554221 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 15:00:04.554323 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 15:00:04.554526 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":535,"threads_started":1}
I20260620 15:00:04.555114 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:04.541007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:44387 (local address 127.2.70.62:42913)
0620 15:00:04.541199 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:04.541207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:04.541862 (+ 655us) server_negotiation.cc:407] Connection header received
0620 15:00:04.542068 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:04.542074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:04.542160 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:04.542277 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:04.543671 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:04.544445 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:04.546063 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:04.546740 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:04.546855 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:04.548921 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:04.548927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:04.548931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:04.548969 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:04.552463 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:04.553076 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:04.553084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:04.553091 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:04.553236 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:04.553592 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:04.553599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:04.553605 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:04.553980 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:04.554126 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:04.554467 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 15:00:04.554708 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:00:04.588575 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42913
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:04.910527 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:04.910866 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:04.911041 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:04.918336 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:04.918497 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:04.918661 4298 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:00:04.918751 4298 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:00:04.918864 4298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:04.928515 4298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:04.929896 4298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:04.931257 4298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:04.944348 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:04.944608 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:04.944967 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:04.945847 4298 server_base.cc:1034] running on GCE node
I20260620 15:00:04.946441 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:04.947431 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:04.948650 4298 hybrid_clock.cc:648] HybridClock initialized: now 1781967604948637 us; error 62 us; skew 500 ppm
I20260620 15:00:04.949004 4298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:04 dist-test-slave-n5vv krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:04.952184 4298 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:00:04.953814 4298 webserver.cc:460] Webserver started at http://127.2.70.2:37513/ using document root <none> and password file <none>
I20260620 15:00:04.954555 4298 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:04.954677 4298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:04.954988 4298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:04.957840 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "c6b9d46d6791470995d7470a7d97d84d"
format_stamp: "Formatted at 2026-06-20 15:00:04 on dist-test-slave-n5vv"
server_key: "775c8e5807431a3f2344cdc644d905fc"
server_key_iv: "249d7a499a4c1993ea3f6d6fb90a2f62"
server_key_version: "encryptionkey@0"
I20260620 15:00:04.958570 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "c6b9d46d6791470995d7470a7d97d84d"
format_stamp: "Formatted at 2026-06-20 15:00:04 on dist-test-slave-n5vv"
server_key: "775c8e5807431a3f2344cdc644d905fc"
server_key_iv: "249d7a499a4c1993ea3f6d6fb90a2f62"
server_key_version: "encryptionkey@0"
I20260620 15:00:04.963675 4298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:04.967236 4313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:04.968565 4298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:00:04.968740 4298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "c6b9d46d6791470995d7470a7d97d84d"
format_stamp: "Formatted at 2026-06-20 15:00:04 on dist-test-slave-n5vv"
server_key: "775c8e5807431a3f2344cdc644d905fc"
server_key_iv: "249d7a499a4c1993ea3f6d6fb90a2f62"
server_key_version: "encryptionkey@0"
I20260620 15:00:04.968941 4298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:04.992496 4298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:04.996922 4298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:04.997231 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:04.999920 4298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:05.000037 4298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:05.000145 4298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:05.000228 4298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:00:05 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967604, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:05.027375 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.000717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:39887 (local address 127.2.70.62:42913)
0620 15:00:05.000897 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.000904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.002680 (+ 1776us) server_negotiation.cc:407] Connection header received
0620 15:00:05.004807 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.004820 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.004921 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.005085 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:05.008076 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.009106 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.012630 (+ 3524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.013158 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.013307 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.019289 (+ 5982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:05.019294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:05.019298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:05.019333 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:05.023783 (+ 4450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.024794 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.024799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.024803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.024962 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.025622 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.025625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.025627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.025894 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:05.026058 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.026796 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.027016 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:00:05.028389 4322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.001665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42913 (local address 127.2.70.2:39887)
0620 15:00:05.002525 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.002562 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.004459 (+ 1897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.005674 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.005693 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.006711 (+ 1018us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:05.007819 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.007862 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.009326 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.009338 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.012371 (+ 3033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.012393 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.013460 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.013471 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.013868 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.014943 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:05.015015 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:05.018974 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:05.023950 (+ 4976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.023963 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.023988 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.024658 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.025121 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.025131 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.025136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.025388 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.026030 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:05.026039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:05.026408 (+ 369us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.026676 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.027243 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":382,"threads_started":1}
I20260620 15:00:05.049386 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.034998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:51613 (local address 127.2.70.62:42913)
0620 15:00:05.035155 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.035159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.035791 (+ 632us) server_negotiation.cc:407] Connection header received
0620 15:00:05.035991 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.036000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.036126 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.036243 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:05.037195 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.038723 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.040088 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.040842 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.041848 (+ 1006us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.043727 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:05.043736 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:05.043743 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:05.043785 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:05.046825 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.047479 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.047484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.047488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.047601 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.047946 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.047953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.047956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.048270 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:05.048417 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.048812 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.049038 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:00:05.049536 4397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.035028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42913 (local address 127.2.70.2:51613)
0620 15:00:05.035668 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.035688 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.035817 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.036253 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.036256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.036712 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:05.037058 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.037068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.038870 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.038874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.039951 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.039961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.041105 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.041117 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.041390 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.042223 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:05.042268 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:05.043447 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:05.046957 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.046961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.046965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.047355 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.047706 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.047710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.047714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.047843 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.048385 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:05.048390 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:05.048574 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.048730 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.049050 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":342,"threads_started":1}
I20260620 15:00:05.055860 4298 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:44907
I20260620 15:00:05.055998 4432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:44907 every 8 connection(s)
I20260620 15:00:05.057087 4298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:05.057469 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 4298
I20260620 15:00:05.057680 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:05.058310 2328 external_mini_cluster.cc:1428] Setting key 5d76a4722d693015096ee7ec6ef32fd6
I20260620 15:00:05.059283 4433 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42913
I20260620 15:00:05.059530 4433 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:05.060212 4433 heartbeater.cc:511] Master 127.2.70.62:42913 requested a full tablet report, sending...
I20260620 15:00:05.061626 4086 ts_manager.cc:194] Registered new tserver with Master: c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907)
I20260620 15:00:05.062494 4086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:39887
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:00:05.100926 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42913
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:05.430318 4437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:05.430596 4437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:05.430696 4437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:05.437911 4437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:05.438007 4437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:05.438088 4437 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 15:00:05.438139 4437 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:00:05.438199 4437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:05.447754 4437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:42913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:05.448969 4437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:05.450208 4437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:05.463757 4445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:05.463878 4442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:05.463824 4443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:05.464532 4437 server_base.cc:1034] running on GCE node
I20260620 15:00:05.465112 4437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:05.466079 4437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:05.467289 4437 hybrid_clock.cc:648] HybridClock initialized: now 1781967605467246 us; error 28 us; skew 500 ppm
I20260620 15:00:05.467638 4437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:05 dist-test-slave-n5vv krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:05.474002 4437 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:00:05.475571 4437 webserver.cc:460] Webserver started at http://127.2.70.3:34667/ using document root <none> and password file <none>
I20260620 15:00:05.476269 4437 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:05.476405 4437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:05.476719 4437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:05.479611 4437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "c746e03c56c44b2292c3b93fb4946f41"
format_stamp: "Formatted at 2026-06-20 15:00:05 on dist-test-slave-n5vv"
server_key: "2300656d6e5c101042dac8e8d6eed78e"
server_key_iv: "3e3b41a9043d14196e3dd18f7bd0b1a9"
server_key_version: "encryptionkey@0"
I20260620 15:00:05.480357 4437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "c746e03c56c44b2292c3b93fb4946f41"
format_stamp: "Formatted at 2026-06-20 15:00:05 on dist-test-slave-n5vv"
server_key: "2300656d6e5c101042dac8e8d6eed78e"
server_key_iv: "3e3b41a9043d14196e3dd18f7bd0b1a9"
server_key_version: "encryptionkey@0"
I20260620 15:00:05.485446 4437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:00:05.489099 4452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:05.490696 4437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:05.490885 4437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "c746e03c56c44b2292c3b93fb4946f41"
format_stamp: "Formatted at 2026-06-20 15:00:05 on dist-test-slave-n5vv"
server_key: "2300656d6e5c101042dac8e8d6eed78e"
server_key_iv: "3e3b41a9043d14196e3dd18f7bd0b1a9"
server_key_version: "encryptionkey@0"
I20260620 15:00:05.491099 4437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:05.518456 4437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:05.522466 4437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:05.522779 4437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:05.525645 4437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:05.525791 4437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:05.525930 4437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:05.526059 4437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:00:05 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967605, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:05.546768 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.526091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:40173 (local address 127.2.70.62:42913)
0620 15:00:05.526260 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.526265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.527778 (+ 1513us) server_negotiation.cc:407] Connection header received
0620 15:00:05.530108 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.530114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.530252 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.530380 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:05.532761 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.534125 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.536110 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.536789 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.536926 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.541014 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:05.541019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:05.541022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:05.541052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:05.543694 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.544387 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.544399 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.544402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.544525 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.544915 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.544921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.544925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.545184 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:05.545337 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.546199 (+ 862us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.546409 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:00:05.547808 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.526790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42913 (local address 127.2.70.3:40173)
0620 15:00:05.527625 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.527661 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.529811 (+ 2150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.530685 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.530704 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.531629 (+ 925us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:05.532535 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.532571 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.534317 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.534321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.535941 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.535958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.537029 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.537036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.537239 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.538012 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:05.538050 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:05.540808 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:05.543828 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.543837 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.543851 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.544263 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.544636 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.544642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.544646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.544810 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.545322 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:05.545329 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:05.545795 (+ 466us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.546078 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.546586 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":384,"threads_started":1}
I20260620 15:00:05.551468 4294 heartbeater.cc:503] Master 127.2.70.62:42913 was elected leader, sending a full tablet report...
I20260620 15:00:05.579464 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.561058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:39541 (local address 127.2.70.62:42913)
0620 15:00:05.561230 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.561238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.562065 (+ 827us) server_negotiation.cc:407] Connection header received
0620 15:00:05.562278 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.562286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.562391 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.562504 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:05.564346 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.565600 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.568120 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.568530 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.568631 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.571750 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:05.571759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:05.571766 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:05.571814 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:05.575412 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.576679 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.576688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.576694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.576866 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.577429 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.577437 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.577443 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.577891 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:05.578065 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.578866 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.579165 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 15:00:05.579795 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.560613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42913 (local address 127.2.70.3:39541)
0620 15:00:05.561938 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.561960 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.562140 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.562574 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.562582 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.563478 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:05.564153 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.564170 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.565764 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.565773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.567941 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.567960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.568742 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.568756 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.569047 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.569926 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:05.569957 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:05.571518 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:05.575575 (+ 4057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.575588 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.575598 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.576497 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.577074 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.577081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.577090 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.577292 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.578055 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:05.578063 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:05.578446 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.578707 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.579239 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":1049,"threads_started":1}
I20260620 15:00:05.583657 4437 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:40947
I20260620 15:00:05.583675 4571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:40947 every 8 connection(s)
I20260620 15:00:05.585069 4437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:05.587221 4572 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42913
I20260620 15:00:05.587467 4572 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:05.588102 4572 heartbeater.cc:511] Master 127.2.70.62:42913 requested a full tablet report, sending...
I20260620 15:00:05.589449 4086 ts_manager.cc:194] Registered new tserver with Master: c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:05.590124 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 4437
I20260620 15:00:05.590283 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:05.590533 4086 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:40173
I20260620 15:00:05.590756 2328 external_mini_cluster.cc:1428] Setting key 092a4f4744763a3a68f0e2c2fcc4fda4
I20260620 15:00:05.593612 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:00:05 dist-test-slave-n5vv krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:00:05 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:05.640389 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.626691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.2.70.62:42913)
0620 15:00:05.626857 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.626861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.627145 (+ 284us) server_negotiation.cc:407] Connection header received
0620 15:00:05.627300 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.627304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.627365 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.627453 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:05.628764 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.629459 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.630870 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.631324 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.631417 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.635224 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:05.635230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:05.635233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:05.635270 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:05.638194 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.638763 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.638766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.638769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.638870 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.639198 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.639200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.639202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.639446 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:05.639607 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.639847 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.640001 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:00:05.644264 4086 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:05.651147 4086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:05.687922 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.663303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.0.0.1:40212)
0620 15:00:05.678331 (+ 15028us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.678383 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.678600 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.679799 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.679810 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.679851 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:05.680447 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.680465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.682716 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.682721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.684487 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.684499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.686806 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.686836 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.687051 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.687101 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.687244 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.687441 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14751,"spinlock_wait_cycles":3328,"thread_start_us":15438,"threads_started":1}
I20260620 15:00:05.688074 4588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.663587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.2.70.1:43091)
0620 15:00:05.678682 (+ 15095us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.678692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.678732 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:00:05.678826 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.678833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.679069 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.679306 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:05.680834 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.682536 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.684648 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.686302 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.686433 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.686660 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.687580 (+ 920us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.687760 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14956,"thread_start_us":15882,"threads_started":1}
I20260620 15:00:05.688540 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.663105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.2.70.3:40947)
0620 15:00:05.678456 (+ 15351us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.678468 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.679976 (+ 1508us) server_negotiation.cc:407] Connection header received
0620 15:00:05.680105 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.680117 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.680447 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.680663 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:05.681346 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.683006 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.685770 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.687070 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.687227 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.687454 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.688023 (+ 569us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.688215 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15126,"thread_start_us":15378,"threads_started":1}
I20260620 15:00:05.689568 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.662753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:40947 (local address 127.0.0.1:43264)
0620 15:00:05.679298 (+ 16545us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.679314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.679425 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.680652 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.680657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.680681 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:05.681086 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.681100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.683174 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.683181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.685548 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.685564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.687300 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.687307 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.687553 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.687589 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.687672 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.687799 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15244,"mutex_wait_us":50,"thread_start_us":15489,"threads_started":1}
I20260620 15:00:05.690558 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.679380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.2.70.2:44907)
0620 15:00:05.682471 (+ 3091us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.682491 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.682519 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:00:05.682605 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.682613 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.682838 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.683869 (+ 1031us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:05.684067 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.685520 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.688159 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.689367 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.689536 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.689742 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.690131 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.690293 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2677,"thread_start_us":2323,"threads_started":1}
I20260620 15:00:05.692171 4224 tablet_service.cc:1467] Processing CreateTablet for tablet c2ea9959a3c14617b43d156739eac753 (DEFAULT_TABLE table=test-table [id=d7e9caca2a6e454aafe3d53a0aca1ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:05.692936 4506 tablet_service.cc:1467] Processing CreateTablet for tablet c2ea9959a3c14617b43d156739eac753 (DEFAULT_TABLE table=test-table [id=d7e9caca2a6e454aafe3d53a0aca1ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:05.693377 4224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ea9959a3c14617b43d156739eac753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:05.693881 4589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.679109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:44907 (local address 127.0.0.1:42018)
0620 15:00:05.681355 (+ 2246us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.681373 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.681513 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.683048 (+ 1535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.683055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.683087 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:05.683636 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.683649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.685663 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.685668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.687968 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.687983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.689619 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.689627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.689812 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.689843 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.689951 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.693520 (+ 3569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2092,"mutex_wait_us":1,"thread_start_us":5529,"threads_started":1}
I20260620 15:00:05.694217 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ea9959a3c14617b43d156739eac753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:05.697153 4363 tablet_service.cc:1467] Processing CreateTablet for tablet c2ea9959a3c14617b43d156739eac753 (DEFAULT_TABLE table=test-table [id=d7e9caca2a6e454aafe3d53a0aca1ef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:05.698436 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ea9959a3c14617b43d156739eac753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:05.705425 4591 tablet_bootstrap.cc:492] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Bootstrap starting.
I20260620 15:00:05.707805 4592 tablet_bootstrap.cc:492] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Bootstrap starting.
I20260620 15:00:05.709748 4591 tablet_bootstrap.cc:654] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:05.711076 4592 tablet_bootstrap.cc:654] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:05.711158 4591 log.cc:824] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:05.712218 4592 log.cc:824] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:05.713308 4594 tablet_bootstrap.cc:492] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Bootstrap starting.
I20260620 15:00:05.714143 4591 tablet_bootstrap.cc:492] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: No bootstrap required, opened a new log
I20260620 15:00:05.714392 4591 ts_tablet_manager.cc:1397] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260620 15:00:05.715159 4592 tablet_bootstrap.cc:492] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: No bootstrap required, opened a new log
I20260620 15:00:05.715397 4592 ts_tablet_manager.cc:1397] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 15:00:05.716537 4594 tablet_bootstrap.cc:654] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:05.717715 4594 log.cc:824] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:05.720554 4594 tablet_bootstrap.cc:492] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: No bootstrap required, opened a new log
I20260620 15:00:05.720796 4594 ts_tablet_manager.cc:1397] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:00:05.722200 4591 raft_consensus.cc:348] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.722565 4591 raft_consensus.cc:374] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:05.722662 4591 raft_consensus.cc:729] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c746e03c56c44b2292c3b93fb4946f41, State: Initialized, Role: FOLLOWER
I20260620 15:00:05.723308 4591 consensus_queue.cc:260] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.725572 4592 raft_consensus.cc:348] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.725991 4592 raft_consensus.cc:374] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:05.726086 4592 raft_consensus.cc:729] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Initialized, Role: FOLLOWER
I20260620 15:00:05.726281 4572 heartbeater.cc:503] Master 127.2.70.62:42913 was elected leader, sending a full tablet report...
I20260620 15:00:05.726655 4592 consensus_queue.cc:260] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.728315 4592 ts_tablet_manager.cc:1428] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260620 15:00:05.730397 4591 ts_tablet_manager.cc:1428] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Time spent starting tablet: real 0.016s user 0.008s sys 0.005s
I20260620 15:00:05.731220 4594 raft_consensus.cc:348] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.731714 4594 raft_consensus.cc:374] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:05.731827 4594 raft_consensus.cc:729] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b9d46d6791470995d7470a7d97d84d, State: Initialized, Role: FOLLOWER
I20260620 15:00:05.732491 4594 consensus_queue.cc:260] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.733465 4433 heartbeater.cc:503] Master 127.2.70.62:42913 was elected leader, sending a full tablet report...
I20260620 15:00:05.734018 4594 ts_tablet_manager.cc:1428] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260620 15:00:05.735953 4598 raft_consensus.cc:482] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:05.736225 4598 raft_consensus.cc:504] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.737999 4598 leader_election.cc:288] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947), c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907)
I20260620 15:00:05.745416 4586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.738342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:60483 (local address 127.2.70.3:40947)
0620 15:00:05.738585 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.738593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.739125 (+ 532us) server_negotiation.cc:407] Connection header received
0620 15:00:05.739471 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.739475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.739529 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.739628 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:05.740367 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.741454 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.743480 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.744616 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.744710 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.744782 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.745155 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.745241 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:00:05.745630 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.738428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:40947 (local address 127.2.70.1:60483)
0620 15:00:05.739022 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.739038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.739197 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.739647 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.739651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.739677 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:05.740178 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.740193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.741610 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.741615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.743315 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.743326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.744809 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.744816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.744983 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.745019 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.745116 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.745250 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":349,"threads_started":1}
I20260620 15:00:05.746454 4590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.739232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:41301 (local address 127.2.70.2:44907)
0620 15:00:05.739398 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.739403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.740405 (+ 1002us) server_negotiation.cc:407] Connection header received
0620 15:00:05.740579 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.740584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.740644 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.740749 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:05.741305 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.742391 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.744353 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.745419 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.745578 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.745692 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.746178 (+ 486us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.746272 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 15:00:05.746604 4601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.739172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:44907 (local address 127.2.70.1:41301)
0620 15:00:05.740293 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.740311 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.740464 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.740773 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.740777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.740806 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:05.741152 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.741162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.742528 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.742532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.744201 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.744215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.745694 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.745702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.746017 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.746040 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.746154 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.746290 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":788,"threads_started":1}
I20260620 15:00:05.746620 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ea9959a3c14617b43d156739eac753" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41" is_pre_election: true
I20260620 15:00:05.747155 4527 raft_consensus.cc:2454] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 0.
I20260620 15:00:05.747386 4383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ea9959a3c14617b43d156739eac753" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9d46d6791470995d7470a7d97d84d" is_pre_election: true
I20260620 15:00:05.747812 4177 leader_election.cc:302] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed26b54b84b4c33af528ce20bfc0514, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:05.747851 4383 raft_consensus.cc:2454] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 0.
I20260620 15:00:05.748199 4598 raft_consensus.cc:2781] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:05.748349 4598 raft_consensus.cc:482] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:05.748520 4598 raft_consensus.cc:3037] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:05.751010 4598 raft_consensus.cc:504] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.751592 4598 leader_election.cc:288] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 election: Requested vote from peers c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947), c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907)
I20260620 15:00:05.751967 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ea9959a3c14617b43d156739eac753" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41"
I20260620 15:00:05.752036 4383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ea9959a3c14617b43d156739eac753" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9d46d6791470995d7470a7d97d84d"
I20260620 15:00:05.752169 4527 raft_consensus.cc:3037] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:05.752206 4383 raft_consensus.cc:3037] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:05.755371 4383 raft_consensus.cc:2454] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 1.
I20260620 15:00:05.755371 4527 raft_consensus.cc:2454] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 1.
I20260620 15:00:05.755707 4177 leader_election.cc:302] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed26b54b84b4c33af528ce20bfc0514, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:05.755995 4598 raft_consensus.cc:2781] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:05.756482 4598 raft_consensus.cc:686] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Running, Role: LEADER
I20260620 15:00:05.757009 4598 consensus_queue.cc:237] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } }
I20260620 15:00:05.761325 4086 catalog_manager.cc:5495] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } health_report { overall_health: UNKNOWN } } }
W20260620 15:00:05.793839 4295 tablet.cc:2369] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:00:05.809195 4434 tablet.cc:2369] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 15:00:05 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967603, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:05.824704 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.811792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.2.70.62:42913)
0620 15:00:05.812010 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:05.812019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:05.812616 (+ 597us) server_negotiation.cc:407] Connection header received
0620 15:00:05.812752 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:05.812756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:05.812813 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:05.812894 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:05.814005 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.814716 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.815961 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.816360 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.816450 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.819618 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:05.819622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:05.819626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:05.819653 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:05.822176 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.822804 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.822808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.822810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.822939 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:05.823374 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:05.823377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:05.823379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:05.823652 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:05.823805 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:05.824174 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 15:00:05.824452 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":66}
I20260620 15:00:05.824734 4609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:05.811780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42913 (local address 127.0.0.1:38394)
0620 15:00:05.812510 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:05.812528 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:05.812653 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:05.812939 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:05.812942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:05.813614 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:05.813883 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.813892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.814855 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.814859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.815823 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:05.815843 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:05.816577 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:05.816583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:05.816702 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:05.817319 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:05.817335 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:05.819459 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:05.822293 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.822302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.822308 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.822704 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.823097 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:05.823105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:05.823111 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:05.823275 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:05.823764 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:05.823771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:05.823934 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 15:00:05.824091 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 15:00:05.824346 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":441,"threads_started":1}
I20260620 15:00:05.829964 4086 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:38394:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:00:05.830967 4086 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260620 15:00:05.836740 4573 tablet.cc:2369] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:05.837435 4224 tablet_service.cc:1467] Processing CreateTablet for tablet 9850c0af82dd4ed4a27bdd5379da940d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b7e2df3aaf940428882fa98abeb94d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:05.837564 4363 tablet_service.cc:1467] Processing CreateTablet for tablet 9850c0af82dd4ed4a27bdd5379da940d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b7e2df3aaf940428882fa98abeb94d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:05.838321 4224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850c0af82dd4ed4a27bdd5379da940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:05.837873 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 9850c0af82dd4ed4a27bdd5379da940d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3b7e2df3aaf940428882fa98abeb94d7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:05.838451 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850c0af82dd4ed4a27bdd5379da940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:05.838658 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9850c0af82dd4ed4a27bdd5379da940d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:05.843807 4592 tablet_bootstrap.cc:492] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: Bootstrap starting.
I20260620 15:00:05.845544 4592 tablet_bootstrap.cc:654] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:05.846873 4594 tablet_bootstrap.cc:492] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: Bootstrap starting.
I20260620 15:00:05.848173 4592 tablet_bootstrap.cc:492] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: No bootstrap required, opened a new log
I20260620 15:00:05.848315 4592 ts_tablet_manager.cc:1397] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260620 15:00:05.848870 4594 tablet_bootstrap.cc:654] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:05.848939 4592 raft_consensus.cc:348] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:05.849175 4592 raft_consensus.cc:374] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:05.849285 4592 raft_consensus.cc:729] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Initialized, Role: FOLLOWER
I20260620 15:00:05.849558 4592 consensus_queue.cc:260] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:05.850068 4592 ts_tablet_manager.cc:1428] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260620 15:00:05.850435 4591 tablet_bootstrap.cc:492] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: Bootstrap starting.
I20260620 15:00:05.850394 4598 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:05.850780 4598 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:05.851749 4591 tablet_bootstrap.cc:654] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:05.853884 4591 tablet_bootstrap.cc:492] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: No bootstrap required, opened a new log
I20260620 15:00:05.854029 4591 ts_tablet_manager.cc:1397] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260620 15:00:05.854151 4594 tablet_bootstrap.cc:492] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: No bootstrap required, opened a new log
I20260620 15:00:05.854300 4594 ts_tablet_manager.cc:1397] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260620 15:00:05.854849 4591 raft_consensus.cc:348] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:05.855088 4591 raft_consensus.cc:374] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:05.855185 4591 raft_consensus.cc:729] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c746e03c56c44b2292c3b93fb4946f41, State: Initialized, Role: FOLLOWER
I20260620 15:00:05.855413 4591 consensus_queue.cc:260] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:05.855537 4594 raft_consensus.cc:348] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:05.855801 4594 raft_consensus.cc:374] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:05.855890 4594 raft_consensus.cc:729] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b9d46d6791470995d7470a7d97d84d, State: Initialized, Role: FOLLOWER
I20260620 15:00:05.855985 4591 ts_tablet_manager.cc:1428] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:05.856127 4594 consensus_queue.cc:260] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:05.856384 4597 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:05.856740 4597 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:05.856797 4594 ts_tablet_manager.cc:1428] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 15:00:05.856607 4599 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:05.857028 4599 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:06.197170 4598 consensus_queue.cc:1035] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:06.205313 4602 consensus_queue.cc:1035] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:06.287127 4602 raft_consensus.cc:482] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:06.287312 4602 raft_consensus.cc:504] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.287925 4602 leader_election.cc:288] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907), c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:06.288620 4383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850c0af82dd4ed4a27bdd5379da940d" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9d46d6791470995d7470a7d97d84d" is_pre_election: true
I20260620 15:00:06.288620 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850c0af82dd4ed4a27bdd5379da940d" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41" is_pre_election: true
I20260620 15:00:06.288861 4527 raft_consensus.cc:2454] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 0.
I20260620 15:00:06.288892 4383 raft_consensus.cc:2454] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 0.
I20260620 15:00:06.289355 4177 leader_election.cc:302] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed26b54b84b4c33af528ce20bfc0514, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:06.289685 4602 raft_consensus.cc:2781] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:06.289794 4602 raft_consensus.cc:482] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:06.289924 4602 raft_consensus.cc:3037] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:06.291886 4602 raft_consensus.cc:504] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.292469 4602 leader_election.cc:288] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 election: Requested vote from peers c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907), c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:06.292838 4383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850c0af82dd4ed4a27bdd5379da940d" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9d46d6791470995d7470a7d97d84d"
I20260620 15:00:06.293053 4383 raft_consensus.cc:3037] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:06.292979 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9850c0af82dd4ed4a27bdd5379da940d" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41"
I20260620 15:00:06.293222 4527 raft_consensus.cc:3037] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:06.295226 4527 raft_consensus.cc:2454] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 1.
I20260620 15:00:06.295296 4383 raft_consensus.cc:2454] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 1.
I20260620 15:00:06.295634 4177 leader_election.cc:302] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed26b54b84b4c33af528ce20bfc0514, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:06.295881 4602 raft_consensus.cc:2781] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:06.296070 4602 raft_consensus.cc:686] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Running, Role: LEADER
I20260620 15:00:06.296315 4602 consensus_queue.cc:237] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.297911 4598 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: TxnStatusTablet state changed. Reason: New leader 8ed26b54b84b4c33af528ce20bfc0514. Latest consensus state: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:06.298154 4598 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:06.298703 4619 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:00:06.298939 4619 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:00:06.299214 4086 catalog_manager.cc:5495] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:06.424227 4622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.414399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.0.0.1:40224)
0620 15:00:06.417945 (+ 3546us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:06.417962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:06.418096 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:06.418642 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:06.418646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:06.418656 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:06.418880 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.418889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.420303 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.420307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.421372 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.421385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.422371 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.422377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.422498 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.422554 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.423700 (+ 1146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.423732 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:00:06.423813 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 15:00:06.423975 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3390,"thread_start_us":3275,"threads_started":1}
I20260620 15:00:06.424319 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.414512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.2.70.1:43091)
0620 15:00:06.415106 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:06.415113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:06.418053 (+ 2940us) server_negotiation.cc:407] Connection header received
0620 15:00:06.418288 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:06.418294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:06.418411 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:06.418600 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:06.419024 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.420162 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.421538 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.422227 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.422353 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.422747 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.423566 (+ 819us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.423732 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:06.423877 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 15:00:06.423962 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":270,"threads_started":1}
I20260620 15:00:06.431509 4383 raft_consensus.cc:1261] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Refusing update from remote peer 8ed26b54b84b4c33af528ce20bfc0514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:06.431529 4527 raft_consensus.cc:1261] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Refusing update from remote peer 8ed26b54b84b4c33af528ce20bfc0514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:06.432122 4598 consensus_queue.cc:1035] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:06.432713 4602 consensus_queue.cc:1035] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:06.438313 4597 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: TxnStatusTablet state changed. Reason: New leader 8ed26b54b84b4c33af528ce20bfc0514. Latest consensus state: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:06.438582 4597 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:06.439939 4599 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: TxnStatusTablet state changed. Reason: New leader 8ed26b54b84b4c33af528ce20bfc0514. Latest consensus state: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:06.440176 4599 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:06.444027 4615 mvcc.cc:205] Tried to move back new op lower bound from 7298939315935641600 to 7298939315391926272. Current Snapshot: MvccSnapshot[applied={T|T < 7298939315935641600}]
I20260620 15:00:06.445590 4599 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:06.445887 4599 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P c6b9d46d6791470995d7470a7d97d84d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:06.450166 4597 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:06.450428 4597 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P c746e03c56c44b2292c3b93fb4946f41: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:06.462564 4598 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:06.462821 4598 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:06.463020 4602 tablet_replica.cc:439] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } } }
I20260620 15:00:06.463299 4602 tablet_replica.cc:442] T 9850c0af82dd4ed4a27bdd5379da940d P 8ed26b54b84b4c33af528ce20bfc0514: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:06.468955 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.461664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.2.70.1:43091)
0620 15:00:06.461884 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:06.461893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:06.463930 (+ 2037us) server_negotiation.cc:407] Connection header received
0620 15:00:06.464127 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:06.464134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:06.464217 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:06.464321 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:06.464819 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.465747 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.467285 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.467738 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.467883 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.468281 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.468370 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.468475 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:06.468613 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 15:00:06.468686 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:00:06.488731 4634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.482035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.2.70.1:51439)
0620 15:00:06.482825 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:06.482840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:06.482961 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:06.483291 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:06.483294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:06.483302 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:06.483602 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.483610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.485110 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.485113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.486565 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.486577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.487589 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.487593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.487742 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.487796 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.488219 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.488226 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:00:06.488283 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 15:00:06.488380 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":403,"threads_started":1}
I20260620 15:00:06.488768 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.481913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:51439 (local address 127.2.70.1:43091)
0620 15:00:06.482136 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:06.482142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:06.482934 (+ 792us) server_negotiation.cc:407] Connection header received
0620 15:00:06.483099 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:06.483104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:06.483181 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:06.483281 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:06.483747 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.484942 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.486742 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.487401 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.487551 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.488000 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.488126 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.488212 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:06.488330 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 15:00:06.488400 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:00:06.574911 4087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:06.575203 4087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:06.578886 4087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: Sending DeleteTablet for 3 replicas of tablet c2ea9959a3c14617b43d156739eac753
I20260620 15:00:06.580065 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ea9959a3c14617b43d156739eac753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:06 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:43264
I20260620 15:00:06.580467 4363 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ea9959a3c14617b43d156739eac753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:06 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:42018
I20260620 15:00:06.581138 4637 tablet_replica.cc:330] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: stopping tablet replica
I20260620 15:00:06.581193 4224 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ea9959a3c14617b43d156739eac753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:06 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:40212
I20260620 15:00:06.581738 4637 raft_consensus.cc:2229] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:06.582198 4637 raft_consensus.cc:2258] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:06.582609 4087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:06.583417 4087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:06.584457 4637 ts_tablet_manager.cc:1905] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:06.586680 4638 tablet_replica.cc:330] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: stopping tablet replica
I20260620 15:00:06.587167 4638 raft_consensus.cc:2229] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:06.588876 4638 raft_consensus.cc:2258] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:06.591270 4223 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 (DEFAULT_TABLE table=test-table [id=493dd3864cdb49db821ca631fa8739cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:06.591508 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 (DEFAULT_TABLE table=test-table [id=493dd3864cdb49db821ca631fa8739cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:06.591816 4223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:06.591930 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:06.594128 4637 ts_tablet_manager.cc:1918] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:06.594337 4637 log.cc:1196] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/c2ea9959a3c14617b43d156739eac753
I20260620 15:00:06.594961 4637 ts_tablet_manager.cc:1939] T c2ea9959a3c14617b43d156739eac753 P c746e03c56c44b2292c3b93fb4946f41: Deleting consensus metadata
I20260620 15:00:06.598063 4074 catalog_manager.cc:4841] TS c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947): tablet c2ea9959a3c14617b43d156739eac753 (table test-table [id=d7e9caca2a6e454aafe3d53a0aca1ef0]) successfully deleted
I20260620 15:00:06.598613 4363 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 (DEFAULT_TABLE table=test-table [id=493dd3864cdb49db821ca631fa8739cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:06.599047 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:06.600319 4640 tablet_bootstrap.cc:492] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: Bootstrap starting.
I20260620 15:00:06.601366 4638 ts_tablet_manager.cc:1905] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:06.603120 4640 tablet_bootstrap.cc:654] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:06.609115 4638 ts_tablet_manager.cc:1918] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:06.609306 4638 log.cc:1196] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/c2ea9959a3c14617b43d156739eac753
I20260620 15:00:06.610507 4638 ts_tablet_manager.cc:1939] T c2ea9959a3c14617b43d156739eac753 P c6b9d46d6791470995d7470a7d97d84d: Deleting consensus metadata
I20260620 15:00:06.612396 4074 catalog_manager.cc:4841] TS c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907): tablet c2ea9959a3c14617b43d156739eac753 (table test-table [id=d7e9caca2a6e454aafe3d53a0aca1ef0]) successfully deleted
I20260620 15:00:06.618086 4643 tablet_bootstrap.cc:492] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: Bootstrap starting.
I20260620 15:00:06.620153 4642 tablet_bootstrap.cc:492] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: Bootstrap starting.
I20260620 15:00:06.624661 4643 tablet_bootstrap.cc:654] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:06.627043 4642 tablet_bootstrap.cc:654] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:06.628710 4639 tablet_replica.cc:330] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: stopping tablet replica
I20260620 15:00:06.629213 4639 raft_consensus.cc:2229] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:06.629947 4639 raft_consensus.cc:2258] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:06.630000 4642 tablet_bootstrap.cc:492] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: No bootstrap required, opened a new log
I20260620 15:00:06.630137 4642 ts_tablet_manager.cc:1397] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260620 15:00:06.630950 4642 raft_consensus.cc:348] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.631202 4642 raft_consensus.cc:374] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:06.631313 4642 raft_consensus.cc:729] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b9d46d6791470995d7470a7d97d84d, State: Initialized, Role: FOLLOWER
I20260620 15:00:06.631975 4639 ts_tablet_manager.cc:1905] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:06.632299 4642 consensus_queue.cc:260] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.632911 4640 tablet_bootstrap.cc:492] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: No bootstrap required, opened a new log
I20260620 15:00:06.632911 4642 ts_tablet_manager.cc:1428] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:06.633108 4640 ts_tablet_manager.cc:1397] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.012s
I20260620 15:00:06.634044 4640 raft_consensus.cc:348] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.634354 4640 raft_consensus.cc:374] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:06.634495 4640 raft_consensus.cc:729] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Initialized, Role: FOLLOWER
I20260620 15:00:06.634822 4640 consensus_queue.cc:260] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.636113 4640 ts_tablet_manager.cc:1428] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:06.642833 4643 tablet_bootstrap.cc:492] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: No bootstrap required, opened a new log
I20260620 15:00:06.642997 4643 ts_tablet_manager.cc:1397] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.003s
I20260620 15:00:06.643743 4643 raft_consensus.cc:348] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.643994 4643 raft_consensus.cc:374] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:06.644438 4643 raft_consensus.cc:729] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c746e03c56c44b2292c3b93fb4946f41, State: Initialized, Role: FOLLOWER
I20260620 15:00:06.644704 4643 consensus_queue.cc:260] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.645367 4643 ts_tablet_manager.cc:1428] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 15:00:06.648193 4639 ts_tablet_manager.cc:1918] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:06.648382 4639 log.cc:1196] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/c2ea9959a3c14617b43d156739eac753
I20260620 15:00:06.648974 4639 ts_tablet_manager.cc:1939] T c2ea9959a3c14617b43d156739eac753 P 8ed26b54b84b4c33af528ce20bfc0514: Deleting consensus metadata
I20260620 15:00:06.650599 4073 catalog_manager.cc:4841] TS 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1:43091): tablet c2ea9959a3c14617b43d156739eac753 (table test-table [id=d7e9caca2a6e454aafe3d53a0aca1ef0]) successfully deleted
I20260620 15:00:06.684831 4599 raft_consensus.cc:482] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:06.685122 4599 raft_consensus.cc:504] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.686836 4599 leader_election.cc:288] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1:43091), c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:06.695215 4648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.687734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:59779 (local address 127.2.70.3:40947)
0620 15:00:06.688466 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:06.688473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:06.688996 (+ 523us) server_negotiation.cc:407] Connection header received
0620 15:00:06.689169 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:06.689177 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:06.689250 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:06.689356 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:06.690020 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.691060 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.692976 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.694113 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.694233 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.694372 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:06.694712 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 15:00:06.694848 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":424,"threads_started":1}
I20260620 15:00:06.695413 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.687299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:40947 (local address 127.2.70.2:59779)
0620 15:00:06.688871 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:06.688890 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:06.689028 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:06.689407 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:06.689412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:06.689436 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:06.689841 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.689854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.691190 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.691197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.692830 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.692842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.694322 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.694329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.694531 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.694567 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:00:06.694673 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 15:00:06.694854 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"mutex_wait_us":2,"thread_start_us":1291,"threads_started":1}
I20260620 15:00:06.696134 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab5d7bc6b464f348cbbf68cfcd6b215" candidate_uuid: "c6b9d46d6791470995d7470a7d97d84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41" is_pre_election: true
I20260620 15:00:06.696401 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.687459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:52155 (local address 127.2.70.1:43091)
0620 15:00:06.687675 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:06.687681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:06.688256 (+ 575us) server_negotiation.cc:407] Connection header received
0620 15:00:06.688616 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:06.688624 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:06.688709 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:06.688774 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:06.689630 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.690798 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.693431 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.694819 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.694955 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.695048 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:06.695988 (+ 940us) server_negotiation.cc:299] Negotiation successful
0620 15:00:06.696115 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:00:06.696432 4527 raft_consensus.cc:2454] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b9d46d6791470995d7470a7d97d84d in term 0.
I20260620 15:00:06.696753 4646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.687219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.2.70.2:52155)
0620 15:00:06.688147 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:06.688170 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:06.688408 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:06.688995 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:06.688999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:06.689048 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:06.689424 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.689441 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.690942 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.690949 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.693257 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.693275 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.695190 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.695203 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.695803 (+ 600us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.695841 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:00:06.695944 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:00:06.696064 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":96,"spinlock_wait_cycles":6528,"thread_start_us":590,"threads_started":1}
I20260620 15:00:06.696946 4317 leader_election.cc:302] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b9d46d6791470995d7470a7d97d84d, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:06.697404 4599 raft_consensus.cc:2781] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:06.697546 4599 raft_consensus.cc:482] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:06.697665 4599 raft_consensus.cc:3037] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:06.697775 4244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab5d7bc6b464f348cbbf68cfcd6b215" candidate_uuid: "c6b9d46d6791470995d7470a7d97d84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed26b54b84b4c33af528ce20bfc0514" is_pre_election: true
I20260620 15:00:06.698263 4244 raft_consensus.cc:2454] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b9d46d6791470995d7470a7d97d84d in term 0.
I20260620 15:00:06.700433 4599 raft_consensus.cc:504] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.701136 4599 leader_election.cc:288] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [CANDIDATE]: Term 1 election: Requested vote from peers 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1:43091), c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:06.701673 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab5d7bc6b464f348cbbf68cfcd6b215" candidate_uuid: "c6b9d46d6791470995d7470a7d97d84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41"
I20260620 15:00:06.701717 4244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ab5d7bc6b464f348cbbf68cfcd6b215" candidate_uuid: "c6b9d46d6791470995d7470a7d97d84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed26b54b84b4c33af528ce20bfc0514"
I20260620 15:00:06.701895 4527 raft_consensus.cc:3037] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:06.701934 4244 raft_consensus.cc:3037] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:06.703835 4527 raft_consensus.cc:2454] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b9d46d6791470995d7470a7d97d84d in term 1.
I20260620 15:00:06.704211 4317 leader_election.cc:302] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b9d46d6791470995d7470a7d97d84d, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:06.704506 4599 raft_consensus.cc:2781] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:06.705055 4599 raft_consensus.cc:686] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 1 LEADER]: Becoming Leader. State: Replica: c6b9d46d6791470995d7470a7d97d84d, State: Running, Role: LEADER
I20260620 15:00:06.705258 4244 raft_consensus.cc:2454] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b9d46d6791470995d7470a7d97d84d in term 1.
I20260620 15:00:06.705513 4599 consensus_queue.cc:237] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:06.709234 4087 catalog_manager.cc:5495] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b9d46d6791470995d7470a7d97d84d (127.2.70.2). New cstate: current_term: 1 leader_uuid: "c6b9d46d6791470995d7470a7d97d84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:06.755821 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.737707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.2.70.2:44907)
0620 15:00:06.749760 (+ 12053us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:06.749767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:06.749797 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:00:06.749909 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:06.749915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:06.750035 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:06.750234 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:06.750698 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.751878 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.753169 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.753810 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.753937 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.754332 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.755149 (+ 817us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.755286 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:06.755346 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 15:00:06.755446 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11901,"thread_start_us":11723,"threads_started":1}
I20260620 15:00:06.771016 4623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.765475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:45059 (local address 127.2.70.1:43091)
0620 15:00:06.765659 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:06.765664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:06.766246 (+ 582us) server_negotiation.cc:407] Connection header received
0620 15:00:06.766458 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:06.766462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:06.766543 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:06.766636 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:06.767119 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.767837 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.769129 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.769652 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.769802 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.770226 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.770332 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.770451 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:06.770598 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 15:00:06.770680 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 15:00:06.771081 4654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.765503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.2.70.2:45059)
0620 15:00:06.766143 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:06.766159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:06.766319 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:06.766661 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:06.766666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:06.766677 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:06.766978 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.766988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.767967 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.767973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.768987 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.768998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.769837 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.769845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.770003 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.770053 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.770468 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.770489 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 15:00:06.770574 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 15:00:06.770706 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":366,"threads_started":1}
I20260620 15:00:06.779508 4527 raft_consensus.cc:1261] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Refusing update from remote peer c6b9d46d6791470995d7470a7d97d84d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:06.779865 4244 raft_consensus.cc:1261] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Refusing update from remote peer c6b9d46d6791470995d7470a7d97d84d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:06.780699 4649 consensus_queue.cc:1035] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:06.781098 4599 consensus_queue.cc:1035] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:06.787544 4614 mvcc.cc:205] Tried to move back new op lower bound from 7298939317363208192 to 7298939317070020608. Current Snapshot: MvccSnapshot[applied={T|T < 7298939317363208192}]
I20260620 15:00:06.807924 4651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.802203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:50393 (local address 127.2.70.2:44907)
0620 15:00:06.802356 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:06.802362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:06.802379 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:00:06.802440 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:06.802446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:06.802533 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:06.803042 (+ 509us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:06.803208 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.804162 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.806127 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.806779 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.806903 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.807318 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.807409 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.807473 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:06.807659 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 15:00:06.807716 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:00:06.808055 4634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:06.801864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:44907 (local address 127.2.70.1:50393)
0620 15:00:06.802048 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:06.802070 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:06.802176 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:06.802672 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:06.802678 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:06.802693 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:06.802907 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.802924 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.804321 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.804327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.805946 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:06.805957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:06.806910 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:06.806916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:06.807113 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:06.807161 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:06.807506 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:06.807517 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:00:06.807597 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:00:06.807739 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 15:00:06.808959 4615 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 15:00:08.419351 4087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:08.419606 4087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:08.424253 4087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: Sending DeleteTablet for 3 replicas of tablet 8ab5d7bc6b464f348cbbf68cfcd6b215
I20260620 15:00:08.425410 4363 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:42018
I20260620 15:00:08.425455 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:43264
I20260620 15:00:08.425400 4224 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:40212
I20260620 15:00:08.438103 4681 tablet_replica.cc:330] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: stopping tablet replica
I20260620 15:00:08.438380 4681 raft_consensus.cc:2229] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:08.438686 4681 raft_consensus.cc:2258] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:08.439935 4681 ts_tablet_manager.cc:1905] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:08.441253 4679 tablet_replica.cc:330] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: stopping tablet replica
I20260620 15:00:08.441547 4679 raft_consensus.cc:2229] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:08.442157 4679 raft_consensus.cc:2258] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:08.443270 4679 ts_tablet_manager.cc:1905] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:08.447381 4680 tablet_replica.cc:330] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: stopping tablet replica
I20260620 15:00:08.447628 4680 raft_consensus.cc:2229] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:08.447877 4680 raft_consensus.cc:2258] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:08.448930 4680 ts_tablet_manager.cc:1905] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:00:08 dist-test-slave-n5vv krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 15:00:08.543915 2328 mini_kdc.cc:317] Time spent kinit for test-user: real 0.117s user 0.002s sys 0.013s
I20260620 15:00:08.545776 4681 ts_tablet_manager.cc:1918] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:00:08.546041 4681 log.cc:1196] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/8ab5d7bc6b464f348cbbf68cfcd6b215
I20260620 15:00:08.546607 4681 ts_tablet_manager.cc:1939] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P 8ed26b54b84b4c33af528ce20bfc0514: Deleting consensus metadata
I20260620 15:00:08.547919 4679 ts_tablet_manager.cc:1918] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:00:08.548065 4679 log.cc:1196] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/8ab5d7bc6b464f348cbbf68cfcd6b215
I20260620 15:00:08.548496 4679 ts_tablet_manager.cc:1939] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c6b9d46d6791470995d7470a7d97d84d: Deleting consensus metadata
I20260620 15:00:08.551920 4073 catalog_manager.cc:4841] TS 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1:43091): tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 (table test-table [id=493dd3864cdb49db821ca631fa8739cb]) successfully deleted
Jun 20 15:00:08 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:08.568780 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:08.553994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.2.70.62:42913)
0620 15:00:08.554737 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:08.554743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:08.554770 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:00:08.554911 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:08.554916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:08.555009 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:08.555120 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:08.556406 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:08.557607 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:08.559140 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:08.559768 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:08.559891 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:08.563876 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:08.563881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:08.563885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:08.563928 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:08.566525 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:08.567132 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:08.567136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:08.567138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:08.567252 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:08.567588 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:08.567591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:08.567593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:08.567847 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:08.568026 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:08.568266 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 15:00:08.568416 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":433,"threads_started":1}
I20260620 15:00:08.572618 4087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:08.573297 4087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:08.580865 4224 tablet_service.cc:1467] Processing CreateTablet for tablet 79de4329c08a49bc9d36263f8bf2370a (DEFAULT_TABLE table=test-table [id=2a4553d3bb2f4d7a95e2d927a3405724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:08.581072 4506 tablet_service.cc:1467] Processing CreateTablet for tablet 79de4329c08a49bc9d36263f8bf2370a (DEFAULT_TABLE table=test-table [id=2a4553d3bb2f4d7a95e2d927a3405724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:08.581077 4363 tablet_service.cc:1467] Processing CreateTablet for tablet 79de4329c08a49bc9d36263f8bf2370a (DEFAULT_TABLE table=test-table [id=2a4553d3bb2f4d7a95e2d927a3405724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:08.581367 4224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79de4329c08a49bc9d36263f8bf2370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:08.581427 4074 catalog_manager.cc:4841] TS c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907): tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 (table test-table [id=493dd3864cdb49db821ca631fa8739cb]) successfully deleted
I20260620 15:00:08.581601 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79de4329c08a49bc9d36263f8bf2370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:08.587162 4690 tablet_bootstrap.cc:492] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: Bootstrap starting.
I20260620 15:00:08.588812 4690 tablet_bootstrap.cc:654] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:08.590521 4691 tablet_bootstrap.cc:492] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: Bootstrap starting.
I20260620 15:00:08.591116 4690 tablet_bootstrap.cc:492] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: No bootstrap required, opened a new log
I20260620 15:00:08.591259 4690 ts_tablet_manager.cc:1397] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:00:08.592087 4690 raft_consensus.cc:348] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.592304 4691 tablet_bootstrap.cc:654] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:08.592368 4690 raft_consensus.cc:374] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:08.592461 4690 raft_consensus.cc:729] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b9d46d6791470995d7470a7d97d84d, State: Initialized, Role: FOLLOWER
I20260620 15:00:08.593128 4690 consensus_queue.cc:260] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.593791 4690 ts_tablet_manager.cc:1428] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260620 15:00:08.604244 4691 tablet_bootstrap.cc:492] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: No bootstrap required, opened a new log
I20260620 15:00:08.604413 4691 ts_tablet_manager.cc:1397] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260620 15:00:08.605247 4691 raft_consensus.cc:348] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.605441 4691 raft_consensus.cc:374] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:08.605557 4691 raft_consensus.cc:729] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Initialized, Role: FOLLOWER
I20260620 15:00:08.605782 4691 consensus_queue.cc:260] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.606300 4691 ts_tablet_manager.cc:1428] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:08.647434 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79de4329c08a49bc9d36263f8bf2370a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:08.647538 4680 ts_tablet_manager.cc:1918] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:00:08.647689 4680 log.cc:1196] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/8ab5d7bc6b464f348cbbf68cfcd6b215
I20260620 15:00:08.648219 4680 ts_tablet_manager.cc:1939] T 8ab5d7bc6b464f348cbbf68cfcd6b215 P c746e03c56c44b2292c3b93fb4946f41: Deleting consensus metadata
I20260620 15:00:08.649619 4074 catalog_manager.cc:4841] TS c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947): tablet 8ab5d7bc6b464f348cbbf68cfcd6b215 (table test-table [id=493dd3864cdb49db821ca631fa8739cb]) successfully deleted
I20260620 15:00:08.653218 4694 tablet_bootstrap.cc:492] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: Bootstrap starting.
I20260620 15:00:08.654667 4694 tablet_bootstrap.cc:654] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:08.656950 4694 tablet_bootstrap.cc:492] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: No bootstrap required, opened a new log
I20260620 15:00:08.657071 4694 ts_tablet_manager.cc:1397] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260620 15:00:08.657768 4694 raft_consensus.cc:348] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.657948 4694 raft_consensus.cc:374] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:08.658028 4694 raft_consensus.cc:729] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c746e03c56c44b2292c3b93fb4946f41, State: Initialized, Role: FOLLOWER
I20260620 15:00:08.658286 4694 consensus_queue.cc:260] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.659094 4694 ts_tablet_manager.cc:1428] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:08.661154 4661 raft_consensus.cc:482] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:08.661366 4661 raft_consensus.cc:504] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.662122 4661 leader_election.cc:288] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907), c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:08.662622 4383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79de4329c08a49bc9d36263f8bf2370a" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9d46d6791470995d7470a7d97d84d" is_pre_election: true
I20260620 15:00:08.662606 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79de4329c08a49bc9d36263f8bf2370a" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41" is_pre_election: true
I20260620 15:00:08.662870 4383 raft_consensus.cc:2454] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 0.
I20260620 15:00:08.662868 4527 raft_consensus.cc:2454] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 0.
I20260620 15:00:08.663367 4177 leader_election.cc:302] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed26b54b84b4c33af528ce20bfc0514, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:08.663683 4661 raft_consensus.cc:2781] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:08.663812 4661 raft_consensus.cc:482] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:08.663910 4661 raft_consensus.cc:3037] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:08.747421 4661 raft_consensus.cc:504] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.748209 4661 leader_election.cc:288] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 election: Requested vote from peers c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907), c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:08.748770 4383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79de4329c08a49bc9d36263f8bf2370a" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9d46d6791470995d7470a7d97d84d"
I20260620 15:00:08.748903 4527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79de4329c08a49bc9d36263f8bf2370a" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41"
I20260620 15:00:08.749019 4383 raft_consensus.cc:3037] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:08.749183 4527 raft_consensus.cc:3037] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:08.751524 4383 raft_consensus.cc:2454] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 1.
I20260620 15:00:08.752002 4177 leader_election.cc:302] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed26b54b84b4c33af528ce20bfc0514, c6b9d46d6791470995d7470a7d97d84d; no voters:
I20260620 15:00:08.752297 4661 raft_consensus.cc:2781] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:08.752460 4661 raft_consensus.cc:686] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Running, Role: LEADER
I20260620 15:00:08.752764 4661 consensus_queue.cc:237] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.753561 4527 raft_consensus.cc:2454] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 1.
I20260620 15:00:08.756086 4087 catalog_manager.cc:5495] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:08.795650 4699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:08.790431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.2.70.1:43091)
0620 15:00:08.791013 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:08.791017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:08.791034 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:00:08.791081 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:08.791084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:08.791140 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:08.791249 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:08.791703 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:08.792531 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:08.793842 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:08.794310 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:08.794446 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:08.794911 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:08.795015 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:08.795117 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:08.795263 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 15:00:08.795335 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":286,"threads_started":1}
I20260620 15:00:08.804701 4527 raft_consensus.cc:1261] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Refusing update from remote peer 8ed26b54b84b4c33af528ce20bfc0514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:08.804775 4383 raft_consensus.cc:1261] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Refusing update from remote peer 8ed26b54b84b4c33af528ce20bfc0514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:08.805392 4658 consensus_queue.cc:1035] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:08.805811 4674 consensus_queue.cc:1035] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:08.851524 4087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:08.851732 4087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:08.854846 4087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: Sending DeleteTablet for 3 replicas of tablet 79de4329c08a49bc9d36263f8bf2370a
I20260620 15:00:08.855849 4363 tablet_service.cc:1514] Processing DeleteTablet for tablet 79de4329c08a49bc9d36263f8bf2370a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:42018
I20260620 15:00:08.855849 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet 79de4329c08a49bc9d36263f8bf2370a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:43264
I20260620 15:00:08.856137 4679 tablet_replica.cc:330] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: stopping tablet replica
I20260620 15:00:08.856133 4680 tablet_replica.cc:330] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: stopping tablet replica
I20260620 15:00:08.856324 4679 raft_consensus.cc:2229] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:08.856328 4680 raft_consensus.cc:2229] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:08.856510 4679 raft_consensus.cc:2258] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:08.856523 4680 raft_consensus.cc:2258] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:08.857296 4224 tablet_service.cc:1514] Processing DeleteTablet for tablet 79de4329c08a49bc9d36263f8bf2370a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:40212
I20260620 15:00:08.857844 4680 ts_tablet_manager.cc:1905] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:08.857909 4087 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 15:00:08.858261 4679 ts_tablet_manager.cc:1905] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:08.858799 4681 tablet_replica.cc:330] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: stopping tablet replica
W20260620 15:00:08.858846 4087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:08.858987 4681 raft_consensus.cc:2229] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:08.859284 4681 raft_consensus.cc:2258] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:08.860299 4681 ts_tablet_manager.cc:1905] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:08.867139 4224 tablet_service.cc:1467] Processing CreateTablet for tablet fbe5bc791d8d4820b866f7ac2343b327 (DEFAULT_TABLE table=test-table [id=0573949fc39f4eb6b088e0302f161bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:08.867589 4506 tablet_service.cc:1467] Processing CreateTablet for tablet fbe5bc791d8d4820b866f7ac2343b327 (DEFAULT_TABLE table=test-table [id=0573949fc39f4eb6b088e0302f161bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:08.867770 4363 tablet_service.cc:1467] Processing CreateTablet for tablet fbe5bc791d8d4820b866f7ac2343b327 (DEFAULT_TABLE table=test-table [id=0573949fc39f4eb6b088e0302f161bf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:08.952140 4506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe5bc791d8d4820b866f7ac2343b327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:08.952148 4363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe5bc791d8d4820b866f7ac2343b327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:08.959874 4690 tablet_bootstrap.cc:492] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d: Bootstrap starting.
I20260620 15:00:08.960361 4224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbe5bc791d8d4820b866f7ac2343b327. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:08.961627 4690 tablet_bootstrap.cc:654] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:08.961998 4694 tablet_bootstrap.cc:492] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: Bootstrap starting.
I20260620 15:00:08.963495 4694 tablet_bootstrap.cc:654] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:08.973426 4690 tablet_bootstrap.cc:492] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d: No bootstrap required, opened a new log
I20260620 15:00:08.973668 4690 ts_tablet_manager.cc:1397] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20260620 15:00:08.974159 4694 tablet_bootstrap.cc:492] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: No bootstrap required, opened a new log
I20260620 15:00:08.974375 4694 ts_tablet_manager.cc:1397] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260620 15:00:08.974466 4690 raft_consensus.cc:348] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.974915 4690 raft_consensus.cc:374] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:08.975009 4690 raft_consensus.cc:729] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b9d46d6791470995d7470a7d97d84d, State: Initialized, Role: FOLLOWER
I20260620 15:00:08.975217 4694 raft_consensus.cc:348] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.975283 4690 consensus_queue.cc:260] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.975560 4694 raft_consensus.cc:374] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:08.975600 4691 tablet_bootstrap.cc:492] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514: Bootstrap starting.
I20260620 15:00:08.975708 4694 raft_consensus.cc:729] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c746e03c56c44b2292c3b93fb4946f41, State: Initialized, Role: FOLLOWER
I20260620 15:00:08.975961 4690 ts_tablet_manager.cc:1428] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:08.975980 4694 consensus_queue.cc:260] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.977259 4694 ts_tablet_manager.cc:1428] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 15:00:08.977277 4691 tablet_bootstrap.cc:654] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:08.979275 4691 tablet_bootstrap.cc:492] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514: No bootstrap required, opened a new log
I20260620 15:00:08.979415 4691 ts_tablet_manager.cc:1397] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260620 15:00:08.980459 4691 raft_consensus.cc:348] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.980700 4691 raft_consensus.cc:374] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:08.980767 4691 raft_consensus.cc:729] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Initialized, Role: FOLLOWER
I20260620 15:00:08.980994 4691 consensus_queue.cc:260] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.981539 4691 ts_tablet_manager.cc:1428] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 15:00:08.989244 4703 raft_consensus.cc:482] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:08.989423 4703 raft_consensus.cc:504] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:08.990120 4703 leader_election.cc:288] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907), c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:08.990720 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe5bc791d8d4820b866f7ac2343b327" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41" is_pre_election: true
I20260620 15:00:08.990881 4383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe5bc791d8d4820b866f7ac2343b327" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9d46d6791470995d7470a7d97d84d" is_pre_election: true
I20260620 15:00:08.991017 4526 raft_consensus.cc:2454] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 0.
I20260620 15:00:08.991169 4383 raft_consensus.cc:2454] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 0.
I20260620 15:00:08.991518 4177 leader_election.cc:302] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed26b54b84b4c33af528ce20bfc0514, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:08.991792 4703 raft_consensus.cc:2781] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:08.991899 4703 raft_consensus.cc:482] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:08.992056 4703 raft_consensus.cc:3037] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:09.052110 4680 ts_tablet_manager.cc:1918] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:09.052142 4681 ts_tablet_manager.cc:1918] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:09.052287 4680 log.cc:1196] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/79de4329c08a49bc9d36263f8bf2370a
I20260620 15:00:09.052466 4681 log.cc:1196] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/79de4329c08a49bc9d36263f8bf2370a
I20260620 15:00:09.052621 4679 ts_tablet_manager.cc:1918] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:09.052780 4680 ts_tablet_manager.cc:1939] T 79de4329c08a49bc9d36263f8bf2370a P c746e03c56c44b2292c3b93fb4946f41: Deleting consensus metadata
I20260620 15:00:09.052800 4679 log.cc:1196] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/79de4329c08a49bc9d36263f8bf2370a
I20260620 15:00:09.053006 4681 ts_tablet_manager.cc:1939] T 79de4329c08a49bc9d36263f8bf2370a P 8ed26b54b84b4c33af528ce20bfc0514: Deleting consensus metadata
I20260620 15:00:09.053212 4679 ts_tablet_manager.cc:1939] T 79de4329c08a49bc9d36263f8bf2370a P c6b9d46d6791470995d7470a7d97d84d: Deleting consensus metadata
I20260620 15:00:09.052065 4703 raft_consensus.cc:504] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:09.053911 4703 leader_election.cc:288] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 election: Requested vote from peers c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907), c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947)
I20260620 15:00:09.054546 4383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe5bc791d8d4820b866f7ac2343b327" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b9d46d6791470995d7470a7d97d84d"
I20260620 15:00:09.054625 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbe5bc791d8d4820b866f7ac2343b327" candidate_uuid: "8ed26b54b84b4c33af528ce20bfc0514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c746e03c56c44b2292c3b93fb4946f41"
I20260620 15:00:09.054795 4383 raft_consensus.cc:3037] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:09.054842 4526 raft_consensus.cc:3037] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:09.057665 4074 catalog_manager.cc:4841] TS c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947): tablet 79de4329c08a49bc9d36263f8bf2370a (table test-table [id=2a4553d3bb2f4d7a95e2d927a3405724]) successfully deleted
I20260620 15:00:09.057693 4073 catalog_manager.cc:4841] TS 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1:43091): tablet 79de4329c08a49bc9d36263f8bf2370a (table test-table [id=2a4553d3bb2f4d7a95e2d927a3405724]) successfully deleted
I20260620 15:00:09.075232 4074 catalog_manager.cc:4841] TS c6b9d46d6791470995d7470a7d97d84d (127.2.70.2:44907): tablet 79de4329c08a49bc9d36263f8bf2370a (table test-table [id=2a4553d3bb2f4d7a95e2d927a3405724]) successfully deleted
I20260620 15:00:09.156011 4526 raft_consensus.cc:2454] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 1.
I20260620 15:00:09.156001 4383 raft_consensus.cc:2454] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ed26b54b84b4c33af528ce20bfc0514 in term 1.
I20260620 15:00:09.156736 4177 leader_election.cc:302] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ed26b54b84b4c33af528ce20bfc0514, c746e03c56c44b2292c3b93fb4946f41; no voters:
I20260620 15:00:09.157116 4703 raft_consensus.cc:2781] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:09.157291 4703 raft_consensus.cc:686] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 LEADER]: Becoming Leader. State: Replica: 8ed26b54b84b4c33af528ce20bfc0514, State: Running, Role: LEADER
I20260620 15:00:09.157600 4703 consensus_queue.cc:237] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } }
I20260620 15:00:09.160313 4085 catalog_manager.cc:5495] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "8ed26b54b84b4c33af528ce20bfc0514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ed26b54b84b4c33af528ce20bfc0514" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:09.188434 4383 raft_consensus.cc:1261] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Refusing update from remote peer 8ed26b54b84b4c33af528ce20bfc0514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:09.188423 4527 raft_consensus.cc:1261] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Refusing update from remote peer 8ed26b54b84b4c33af528ce20bfc0514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:09.189126 4674 consensus_queue.cc:1035] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c746e03c56c44b2292c3b93fb4946f41" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 40947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:09.189391 4703 consensus_queue.cc:1035] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b9d46d6791470995d7470a7d97d84d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 44907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:09.196906 4662 mvcc.cc:205] Tried to move back new op lower bound from 7298939327230685184 to 7298939327111958528. Current Snapshot: MvccSnapshot[applied={T|T < 7298939327230685184}]
W20260620 15:00:09.209111 4659 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 15:00:10.790940 4085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:10.791213 4085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:10.795725 4085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 789308503fe24b73b4d37ff3ddeb6298: Sending DeleteTablet for 3 replicas of tablet fbe5bc791d8d4820b866f7ac2343b327
I20260620 15:00:10.796648 4363 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe5bc791d8d4820b866f7ac2343b327 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:10 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:42018
I20260620 15:00:10.796774 4506 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe5bc791d8d4820b866f7ac2343b327 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:10 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:43264
I20260620 15:00:10.797423 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4158
I20260620 15:00:10.796801 4223 tablet_service.cc:1514] Processing DeleteTablet for tablet fbe5bc791d8d4820b866f7ac2343b327 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:10 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:40212
I20260620 15:00:10.805732 4730 tablet_replica.cc:330] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: stopping tablet replica
I20260620 15:00:10.806006 4730 raft_consensus.cc:2229] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:10.806303 4730 raft_consensus.cc:2258] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:10.806936 4729 tablet_replica.cc:330] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d: stopping tablet replica
I20260620 15:00:10.807606 4730 ts_tablet_manager.cc:1905] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:10.809733 4729 raft_consensus.cc:2229] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:10.810117 4729 raft_consensus.cc:2258] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:10.811229 4729 ts_tablet_manager.cc:1905] T fbe5bc791d8d4820b866f7ac2343b327 P c6b9d46d6791470995d7470a7d97d84d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:10.811472 4731 tablet_replica.cc:330] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514: stopping tablet replica
I20260620 15:00:10.811691 4731 raft_consensus.cc:2229] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:10.812053 4731 raft_consensus.cc:2258] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:10.813002 4731 ts_tablet_manager.cc:1905] T fbe5bc791d8d4820b866f7ac2343b327 P 8ed26b54b84b4c33af528ce20bfc0514: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:00:10 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:00:10.830175 4733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.798673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.2.70.1:43091)
0620 15:00:10.807227 (+ 8554us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:10.807235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:10.807269 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:00:10.807348 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:10.807356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:10.807477 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:10.807629 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:10.809319 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:10.810937 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:10.813052 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:10.813792 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:10.813936 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:10.823411 (+ 9475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:10.823429 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:10.823447 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:10.823491 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:10.826890 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:10.827697 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:10.827710 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:10.827719 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:10.827927 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:10.828365 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:10.828371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:10.828375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:10.828769 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:10.829093 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:10.829424 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 15:00:10.829723 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8378,"thread_start_us":8232,"threads_started":1}
W20260620 15:00:10.830883 4284 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40244
W20260620 15:00:10.831686 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4158: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:10.831898 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 4158
I20260620 15:00:10.846632 4730 ts_tablet_manager.cc:1918] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 15:00:10.846861 4730 log.cc:1196] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/fbe5bc791d8d4820b866f7ac2343b327
I20260620 15:00:10.847358 4730 ts_tablet_manager.cc:1939] T fbe5bc791d8d4820b866f7ac2343b327 P c746e03c56c44b2292c3b93fb4946f41: Deleting consensus metadata
W20260620 15:00:10.862311 4073 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.1:43091 (error 108)
I20260620 15:00:10.862463 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4298
I20260620 15:00:10.865518 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.864231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.0.0.1:40254)
0620 15:00:10.864966 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:10.865173 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":400,"threads_started":1}
W20260620 15:00:10.865914 4073 catalog_manager.cc:4584] TS 8ed26b54b84b4c33af528ce20bfc0514 (127.2.70.1:43091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbe5bc791d8d4820b866f7ac2343b327: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43091: connect: Connection refused (error 111)
Jun 20 15:00:10 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:00:10.879288 4074 catalog_manager.cc:4841] TS c746e03c56c44b2292c3b93fb4946f41 (127.2.70.3:40947): tablet fbe5bc791d8d4820b866f7ac2343b327 (table test-table [id=0573949fc39f4eb6b088e0302f161bf7]) successfully deleted
I20260620 15:00:10.879806 4735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.863610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.2.70.2:44907)
0620 15:00:10.864168 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:10.864173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:10.864192 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:00:10.864257 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:10.864263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:10.864347 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:10.864494 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:10.866636 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:10.867414 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:10.868706 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:10.869103 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:10.869239 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:10.873616 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:10.873633 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:10.873650 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:10.873690 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:10.876738 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:10.877451 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:10.877468 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:10.877470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:10.877762 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:10.878142 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:10.878148 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:10.878151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:10.878609 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:10.878890 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:10.879196 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 15:00:10.879434 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":289,"threads_started":1}
W20260620 15:00:10.881175 4428 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42030
W20260620 15:00:10.882040 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4298: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:10.882225 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 4298
I20260620 15:00:10.919822 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.919277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.0.0.1:40266)
0620 15:00:10.919510 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:10.919669 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 15:00:10.920826 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.920464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.0.0.1:40268)
0620 15:00:10.920613 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:10.920692 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 15:00:10.955992 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4437
I20260620 15:00:10.956271 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.955808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:44907 (local address 127.0.0.1:42034)
0620 15:00:10.956001 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:10.956077 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.2:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
Jun 20 15:00:10 dist-test-slave-n5vv krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967608, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:00:10.972131 4737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.956735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43266 (local address 127.2.70.3:40947)
0620 15:00:10.957266 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:10.957272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:10.957292 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:00:10.957354 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:10.957359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:10.957434 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:10.957564 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:10.958879 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:10.960032 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:10.961213 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:10.961877 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:10.962013 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:10.966398 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:10.966417 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:10.966441 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:10.966488 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:10.969601 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:10.970193 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:10.970201 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:10.970203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:10.970323 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:10.970631 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:10.970634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:10.970636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:10.971064 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:10.971281 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:10.971563 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 15:00:10.971720 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
W20260620 15:00:10.972855 4567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43266
W20260620 15:00:10.973601 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4437: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:10.973757 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 4437
I20260620 15:00:10.981293 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.980782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.0.0.1:40280)
0620 15:00:10.980985 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:10.981083 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 15:00:10.982316 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.981993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43091 (local address 127.0.0.1:40286)
0620 15:00:10.982128 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:10.982198 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 15:00:10.988739 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.988204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:44907 (local address 127.0.0.1:42044)
0620 15:00:10.988423 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:10.988535 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.2:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:00:10.989894 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:10.989429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:44907 (local address 127.0.0.1:42052)
0620 15:00:10.989640 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:10.989738 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.2:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:00:10.992128 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4055
I20260620 15:00:10.993294 4127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:11.109377 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:11.108639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:44907 (local address 127.0.0.1:42066)
0620 15:00:11.108961 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:11.109114 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.2:44907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 15:00:11.109880 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 4055
2026-06-20T15:00:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8868 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:12 dist-test-slave-n5vv krb5kdc[4742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:12 dist-test-slave-n5vv krb5kdc[4742](info): set up 2 sockets
Jun 20 15:00:12 dist-test-slave-n5vv krb5kdc[4742](info): commencing operation
krb5kdc: starting...
W20260620 15:00:13.906235 2328 mini_kdc.cc:121] Time spent starting KDC: real 2.750s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:14 dist-test-slave-n5vv krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:14.101086 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46433
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:46433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:14.428795 4755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:14.429141 4755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:14.429373 4755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:14.437239 4755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:14.437379 4755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:14.437479 4755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:14.437603 4755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:14.437682 4755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:14.447669 4755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:46433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:14.449069 4755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:14.450415 4755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:14.462646 4761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:14.462745 4760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:14.462745 4763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:14.463441 4755 server_base.cc:1034] running on GCE node
I20260620 15:00:14.464262 4755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:14.465860 4755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:14.467067 4755 hybrid_clock.cc:648] HybridClock initialized: now 1781967614466999 us; error 55 us; skew 500 ppm
I20260620 15:00:14.467430 4755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:14 dist-test-slave-n5vv krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:14.470646 4755 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:00:14.472194 4755 webserver.cc:460] Webserver started at http://127.2.70.62:44709/ using document root <none> and password file <none>
I20260620 15:00:14.472899 4755 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:14.473059 4755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:14.473377 4755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:14.476476 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "fbaa528dffa74d08abca3621c5ba18a7"
format_stamp: "Formatted at 2026-06-20 15:00:14 on dist-test-slave-n5vv"
server_key: "fd395473e05caaf0d4eb331e695af6fe"
server_key_iv: "47fe32e17a017023a382417ff12c66b4"
server_key_version: "encryptionkey@0"
I20260620 15:00:14.477432 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "fbaa528dffa74d08abca3621c5ba18a7"
format_stamp: "Formatted at 2026-06-20 15:00:14 on dist-test-slave-n5vv"
server_key: "fd395473e05caaf0d4eb331e695af6fe"
server_key_iv: "47fe32e17a017023a382417ff12c66b4"
server_key_version: "encryptionkey@0"
I20260620 15:00:14.482776 4755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:00:14.486320 4770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:14.487720 4755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 15:00:14.487911 4755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "fbaa528dffa74d08abca3621c5ba18a7"
format_stamp: "Formatted at 2026-06-20 15:00:14 on dist-test-slave-n5vv"
server_key: "fd395473e05caaf0d4eb331e695af6fe"
server_key_iv: "47fe32e17a017023a382417ff12c66b4"
server_key_version: "encryptionkey@0"
I20260620 15:00:14.488168 4755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:14.525149 4755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:14.530702 4755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:14.530947 4755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:14.549453 4755 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:46433
I20260620 15:00:14.549568 4821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:46433 every 8 connection(s)
I20260620 15:00:14.550881 4755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:14.554674 4822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:14.559530 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 4755
I20260620 15:00:14.559736 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:14.560350 2328 external_mini_cluster.cc:1428] Setting key d7137e59ca7680dafec119344370dcd4
I20260620 15:00:14.566495 4822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7: Bootstrap starting.
I20260620 15:00:14.570211 4822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:14.571187 4822 log.cc:824] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:14.574368 4822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7: No bootstrap required, opened a new log
Jun 20 15:00:14 dist-test-slave-n5vv krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:14.585289 4822 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa528dffa74d08abca3621c5ba18a7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46433 } }
I20260620 15:00:14.585799 4822 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:14.585928 4822 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbaa528dffa74d08abca3621c5ba18a7, State: Initialized, Role: FOLLOWER
I20260620 15:00:14.586416 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:14.562402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.2.70.62:46433)
0620 15:00:14.563314 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:14.563333 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:14.563400 (+ 67us) server_negotiation.cc:407] Connection header received
0620 15:00:14.565795 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:14.565822 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:14.566312 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:14.566766 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:14.568047 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.570060 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.571461 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:14.572313 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:14.572488 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:14.577066 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:14.577076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:14.577101 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:14.577162 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:14.581265 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:14.582198 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:14.582207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:14.582222 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:14.582428 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:14.582893 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:14.582898 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:14.582901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:14.583766 (+ 865us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:14.584052 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:14.584470 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 15:00:14.584909 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":403,"threads_started":1}
I20260620 15:00:14.586647 4822 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa528dffa74d08abca3621c5ba18a7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46433 } }
I20260620 15:00:14.586946 4822 raft_consensus.cc:388] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:14.587098 4822 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:14.587278 4822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:14.589989 4822 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa528dffa74d08abca3621c5ba18a7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46433 } }
I20260620 15:00:14.590413 4822 leader_election.cc:302] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fbaa528dffa74d08abca3621c5ba18a7; no voters:
I20260620 15:00:14.590955 4822 leader_election.cc:288] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:14.591089 4827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:14.593268 4822 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:14.597781 4827 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [term 1 LEADER]: Becoming Leader. State: Replica: fbaa528dffa74d08abca3621c5ba18a7, State: Running, Role: LEADER
I20260620 15:00:14.598335 4827 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa528dffa74d08abca3621c5ba18a7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46433 } }
I20260620 15:00:14.600967 4828 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fbaa528dffa74d08abca3621c5ba18a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa528dffa74d08abca3621c5ba18a7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46433 } } }
I20260620 15:00:14.601367 4828 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:14.600935 4829 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbaa528dffa74d08abca3621c5ba18a7. Latest consensus state: current_term: 1 leader_uuid: "fbaa528dffa74d08abca3621c5ba18a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fbaa528dffa74d08abca3621c5ba18a7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46433 } } }
I20260620 15:00:14.601742 4829 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:14.603160 4841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:14.607705 4841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:14.616243 4841 catalog_manager.cc:1347] Generated new cluster ID: 232d8ad5faba41d49a5173da038270aa
I20260620 15:00:14.616359 4841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:14.623631 4841 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:14.625522 4841 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:14.635216 4841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fbaa528dffa74d08abca3621c5ba18a7: Generated new TSK 0
I20260620 15:00:14.636272 4841 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:00:14.692921 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46433
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:15.023224 4848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:15.023535 4848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:15.023710 4848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:15.031395 4848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:15.031520 4848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:15.031706 4848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:15.041755 4848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:15.043078 4848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:15.044438 4848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:15.057760 4853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:15.057801 4854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:15.057838 4856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:15.059413 4848 server_base.cc:1034] running on GCE node
I20260620 15:00:15.060060 4848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:15.061007 4848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:15.062252 4848 hybrid_clock.cc:648] HybridClock initialized: now 1781967615062209 us; error 65 us; skew 500 ppm
I20260620 15:00:15.062597 4848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:15 dist-test-slave-n5vv krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:15.065960 4848 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:00:15.067567 4848 webserver.cc:460] Webserver started at http://127.2.70.1:33465/ using document root <none> and password file <none>
I20260620 15:00:15.068256 4848 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:15.068398 4848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:15.068712 4848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:15.071643 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "052456b6f6df480bbd3d9422f1aee1ab"
format_stamp: "Formatted at 2026-06-20 15:00:15 on dist-test-slave-n5vv"
server_key: "16417f0a3682cd1d51ddad553ebf30bb"
server_key_iv: "c66ac27a10fa21b933c73c0d60a7e94d"
server_key_version: "encryptionkey@0"
I20260620 15:00:15.072439 4848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "052456b6f6df480bbd3d9422f1aee1ab"
format_stamp: "Formatted at 2026-06-20 15:00:15 on dist-test-slave-n5vv"
server_key: "16417f0a3682cd1d51ddad553ebf30bb"
server_key_iv: "c66ac27a10fa21b933c73c0d60a7e94d"
server_key_version: "encryptionkey@0"
I20260620 15:00:15.077574 4848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:00:15.081116 4863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.082569 4848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:00:15.082746 4848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "052456b6f6df480bbd3d9422f1aee1ab"
format_stamp: "Formatted at 2026-06-20 15:00:15 on dist-test-slave-n5vv"
server_key: "16417f0a3682cd1d51ddad553ebf30bb"
server_key_iv: "c66ac27a10fa21b933c73c0d60a7e94d"
server_key_version: "encryptionkey@0"
I20260620 15:00:15.082955 4848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:15.103044 4848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:15.107630 4848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:15.107928 4848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:15.108912 4848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:15.110292 4848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:15.110365 4848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.110535 4848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:15.110597 4848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.143260 4848 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:44369
I20260620 15:00:15.143306 4975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:44369 every 8 connection(s)
I20260620 15:00:15.144589 4848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:15.151948 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 4848
I20260620 15:00:15.152153 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:15.152932 2328 external_mini_cluster.cc:1428] Setting key 3c6b55201ca8e7377bf7877f14951a91
Jun 20 15:00:15 dist-test-slave-n5vv krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:15.166985 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:15.147153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:45383 (local address 127.2.70.62:46433)
0620 15:00:15.147603 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:15.147608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:15.148717 (+ 1109us) server_negotiation.cc:407] Connection header received
0620 15:00:15.150607 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:15.150611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:15.150676 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:15.150774 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:15.152772 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.153641 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.155715 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.156191 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.156297 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:15.160649 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:15.160654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:15.160658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:15.160690 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:15.163482 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:15.164235 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:15.164241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:15.164245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:15.164364 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:15.164854 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:15.164860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:15.164864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:15.165140 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:15.165427 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:15.166383 (+ 956us) server_negotiation.cc:299] Negotiation successful
0620 15:00:15.166611 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":215,"threads_started":1}
I20260620 15:00:15.168155 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:15.147794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46433 (local address 127.2.70.1:45383)
0620 15:00:15.148547 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:15.148591 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:15.150309 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:15.151005 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:15.151013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:15.151711 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:15.152570 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:15.152598 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.153822 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.153827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:15.155559 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:15.155575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.156439 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.156450 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:15.156801 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:15.157607 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:15.157642 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:15.160390 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:15.163636 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:15.163646 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:15.163666 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:15.164101 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:15.164540 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:15.164544 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:15.164546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:15.164739 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:15.165400 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:15.165409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:15.165956 (+ 547us) client_negotiation.cc:769] Sending connection context
0620 15:00:15.166267 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 15:00:15.166866 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":304,"threads_started":1}
I20260620 15:00:15.169888 4976 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46433
I20260620 15:00:15.170207 4976 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:15.170931 4976 heartbeater.cc:511] Master 127.2.70.62:46433 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:00:15.173316 4787 ts_manager.cc:194] Registered new tserver with Master: 052456b6f6df480bbd3d9422f1aee1ab (127.2.70.1:44369)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
I20260620 15:00:15.175313 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:45383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:00:15.197666 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46433
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:15.518528 4982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:15.518805 4982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:15.518965 4982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:15.526111 4982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:15.526214 4982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:15.526434 4982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:15.536309 4982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:15.537686 4982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:15.539050 4982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:15.551605 4987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:15.551630 4988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:15.552392 4990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:15.552812 4982 server_base.cc:1034] running on GCE node
I20260620 15:00:15.553527 4982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:15.554486 4982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:15.555708 4982 hybrid_clock.cc:648] HybridClock initialized: now 1781967615555650 us; error 50 us; skew 500 ppm
I20260620 15:00:15.556061 4982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:15 dist-test-slave-n5vv krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:15.559379 4982 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:00:15.560910 4982 webserver.cc:460] Webserver started at http://127.2.70.2:43247/ using document root <none> and password file <none>
I20260620 15:00:15.561703 4982 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:15.561849 4982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:15.562163 4982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:15.564988 4982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "67a829c3e871418d9f9d1aa0f0a0335a"
format_stamp: "Formatted at 2026-06-20 15:00:15 on dist-test-slave-n5vv"
server_key: "016865338c36e69fe55e9e08b926b74a"
server_key_iv: "628c6f872354aa8b6bfc28fbc790dbf0"
server_key_version: "encryptionkey@0"
I20260620 15:00:15.565809 4982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "67a829c3e871418d9f9d1aa0f0a0335a"
format_stamp: "Formatted at 2026-06-20 15:00:15 on dist-test-slave-n5vv"
server_key: "016865338c36e69fe55e9e08b926b74a"
server_key_iv: "628c6f872354aa8b6bfc28fbc790dbf0"
server_key_version: "encryptionkey@0"
I20260620 15:00:15.570909 4982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:00:15.574352 4997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.575728 4982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 15:00:15.575946 4982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "67a829c3e871418d9f9d1aa0f0a0335a"
format_stamp: "Formatted at 2026-06-20 15:00:15 on dist-test-slave-n5vv"
server_key: "016865338c36e69fe55e9e08b926b74a"
server_key_iv: "628c6f872354aa8b6bfc28fbc790dbf0"
server_key_version: "encryptionkey@0"
I20260620 15:00:15.576154 4982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:15.594015 4982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:15.598416 4982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:15.598693 4982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:15.599632 4982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:15.600950 4982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:15.601050 4982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.601197 4982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:15.601269 4982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:15.633956 4982 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:36011
I20260620 15:00:15.634006 5109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:36011 every 8 connection(s)
I20260620 15:00:15.635249 4982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:15.645144 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 4982
I20260620 15:00:15.645305 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:15.645893 2328 external_mini_cluster.cc:1428] Setting key 2b424f19a61cccb5cf74b422930c9d60
Jun 20 15:00:15 dist-test-slave-n5vv krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967615, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:15.656282 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:15.638032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:34513 (local address 127.2.70.62:46433)
0620 15:00:15.638230 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:15.638236 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:15.639518 (+ 1282us) server_negotiation.cc:407] Connection header received
0620 15:00:15.641204 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:15.641208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:15.641272 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:15.641373 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:15.642884 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.643572 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.645086 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.645800 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.645941 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:15.649882 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:15.649891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:15.649899 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:15.649944 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:15.653348 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:15.654095 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:15.654104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:15.654107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:15.654235 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:15.654660 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:15.654665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:15.654668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:15.654932 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:15.655080 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:15.655741 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 15:00:15.655953 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:00:15.657243 5112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:15.638674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46433 (local address 127.2.70.2:34513)
0620 15:00:15.639354 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:15.639398 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:15.640958 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:15.641674 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:15.641682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:15.642193 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:15.642733 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:15.642749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.643702 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.643706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:15.644940 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:15.644951 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:15.646065 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:15.646080 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:15.646366 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:15.647096 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:15.647122 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:15.649672 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:15.653524 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:15.653536 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:15.653559 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:15.653971 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:15.654346 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:15.654351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:15.654354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:15.654540 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:15.655057 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:15.655065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:15.655386 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 15:00:15.655639 (+ 253us) client_negotiation.cc:240] Negotiation successful
0620 15:00:15.656143 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":267,"threads_started":1}
I20260620 15:00:15.658943 5110 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46433
I20260620 15:00:15.659283 5110 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:15.660076 5110 heartbeater.cc:511] Master 127.2.70.62:46433 requested a full tablet report, sending...
I20260620 15:00:15.661581 4787 ts_manager.cc:194] Registered new tserver with Master: 67a829c3e871418d9f9d1aa0f0a0335a (127.2.70.2:36011)
I20260620 15:00:15.662401 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:34513
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:00:15.688946 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46433
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:16.008085 5115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:16.008376 5115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:16.008471 5115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:16.015694 5115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:16.015801 5115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:16.015913 5115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:16.025794 5115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:16.027118 5115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:16.028378 5115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:16.041782 5120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:16.041837 5123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:16.041891 5121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:16.042814 5115 server_base.cc:1034] running on GCE node
I20260620 15:00:16.043432 5115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:16.044359 5115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:16.045588 5115 hybrid_clock.cc:648] HybridClock initialized: now 1781967616045523 us; error 50 us; skew 500 ppm
I20260620 15:00:16.045873 5115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:16 dist-test-slave-n5vv krb5kdc[4742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967616, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:16.053288 5115 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:00:16.054903 5115 webserver.cc:460] Webserver started at http://127.2.70.3:41477/ using document root <none> and password file <none>
I20260620 15:00:16.055676 5115 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:16.055821 5115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:16.056136 5115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:16.059127 5115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "680a00bbc5ef4d18be1e5cefd99e3694"
format_stamp: "Formatted at 2026-06-20 15:00:16 on dist-test-slave-n5vv"
server_key: "bec0fe2f24ea3e42470ae4a8bdb96d51"
server_key_iv: "9048a61ef2bf6ad37cd0d8795ddee14d"
server_key_version: "encryptionkey@0"
I20260620 15:00:16.059963 5115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "680a00bbc5ef4d18be1e5cefd99e3694"
format_stamp: "Formatted at 2026-06-20 15:00:16 on dist-test-slave-n5vv"
server_key: "bec0fe2f24ea3e42470ae4a8bdb96d51"
server_key_iv: "9048a61ef2bf6ad37cd0d8795ddee14d"
server_key_version: "encryptionkey@0"
I20260620 15:00:16.065193 5115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:00:16.068676 5130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:16.070179 5115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:16.070356 5115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "680a00bbc5ef4d18be1e5cefd99e3694"
format_stamp: "Formatted at 2026-06-20 15:00:16 on dist-test-slave-n5vv"
server_key: "bec0fe2f24ea3e42470ae4a8bdb96d51"
server_key_iv: "9048a61ef2bf6ad37cd0d8795ddee14d"
server_key_version: "encryptionkey@0"
I20260620 15:00:16.070574 5115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:16.082226 5115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:16.086651 5115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:16.086953 5115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:16.088042 5115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:16.089367 5115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:16.089475 5115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:16.089655 5115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:16.089728 5115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:16.123929 5115 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:36695
I20260620 15:00:16.123977 5242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:36695 every 8 connection(s)
I20260620 15:00:16.125200 5115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:16.126825 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 5115
I20260620 15:00:16.127007 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:16.127440 2328 external_mini_cluster.cc:1428] Setting key 94ead4050ec014686d20ce829793477b
Jun 20 15:00:16 dist-test-slave-n5vv krb5kdc[4742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967616, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:16.145840 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.127909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:58091 (local address 127.2.70.62:46433)
0620 15:00:16.128080 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.128085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.129602 (+ 1517us) server_negotiation.cc:407] Connection header received
0620 15:00:16.131656 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.131660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.131728 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.131817 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:16.133215 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.134049 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.135562 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.135959 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.136053 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.139486 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:16.139491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:16.139494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:16.139525 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:16.142765 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:16.143583 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:16.143587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:16.143589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:16.143713 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:16.144222 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:16.144225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:16.144227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:16.144464 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:16.144565 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:16.145398 (+ 833us) server_negotiation.cc:299] Negotiation successful
0620 15:00:16.145615 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:00:16.147145 5245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:16.128505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46433 (local address 127.2.70.3:58091)
0620 15:00:16.129360 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:16.129502 (+ 142us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:16.131359 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:16.131990 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:16.131998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:16.132504 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:16.133071 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.133087 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.134186 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.134190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.135376 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:16.135388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:16.136135 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:16.136139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:16.136324 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:16.137021 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:16.137042 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:16.139288 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:16.142878 (+ 3590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:16.142886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:16.142902 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:16.143451 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:16.143825 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:16.143831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:16.143837 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:16.144111 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:16.144586 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:16.144594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:16.144996 (+ 402us) client_negotiation.cc:769] Sending connection context
0620 15:00:16.145310 (+ 314us) client_negotiation.cc:240] Negotiation successful
0620 15:00:16.145843 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":338,"threads_started":1}
I20260620 15:00:16.148518 5243 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46433
I20260620 15:00:16.148841 5243 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:16.149565 5243 heartbeater.cc:511] Master 127.2.70.62:46433 requested a full tablet report, sending...
I20260620 15:00:16.150902 4787 ts_manager.cc:194] Registered new tserver with Master: 680a00bbc5ef4d18be1e5cefd99e3694 (127.2.70.3:36695)
I20260620 15:00:16.151696 4787 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:58091
I20260620 15:00:16.155570 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:16.178426 4976 heartbeater.cc:503] Master 127.2.70.62:46433 was elected leader, sending a full tablet report...
W20260620 15:00:16.185940 4978 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:16.183505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.2.70.62:46433)
0620 15:00:16.183682 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:16.183687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:16.184193 (+ 506us) server_negotiation.cc:407] Connection header received
0620 15:00:16.184349 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:16.184353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:16.184451 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:16.184541 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:16.185745 (+ 1204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55344: BlockingRecv error: recv got EOF from 127.0.0.1:55344 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:00:16.189320 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4848
W20260620 15:00:16.192368 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4848: Not authorized: Client connection negotiation failed: client connection to 127.2.70.1:44369: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:00:16.192665 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 4848
I20260620 15:00:16.205852 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4982
W20260620 15:00:16.208503 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4982: Not authorized: Client connection negotiation failed: client connection to 127.2.70.2:36011: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:00:16.208660 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 4982
I20260620 15:00:16.224331 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5115
W20260620 15:00:16.227289 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5115: Not authorized: Client connection negotiation failed: client connection to 127.2.70.3:36695: server requires authentication, but client does not have Kerberos credentials available
I20260620 15:00:16.227510 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 5115
I20260620 15:00:16.242870 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 4755
I20260620 15:00:16.244060 4817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:16.345232 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 4755
2026-06-20T15:00:16Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5231 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:16 dist-test-slave-n5vv krb5kdc[5260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:16 dist-test-slave-n5vv krb5kdc[5260](info): set up 2 sockets
Jun 20 15:00:16 dist-test-slave-n5vv krb5kdc[5260](info): commencing operation
krb5kdc: starting...
W20260620 15:00:17.446561 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:17 dist-test-slave-n5vv krb5kdc[5260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:17.641443 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:33891
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:33891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:17.960131 5273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:17.960403 5273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:17.960566 5273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:17.967782 5273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:17.967880 5273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:17.967974 5273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:17.968075 5273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:17.968165 5273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:17.977871 5273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:33891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:33891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:17.979153 5273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:17.980511 5273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:17.991945 5278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:17.992097 5281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:17.992131 5279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:17.993803 5273 server_base.cc:1034] running on GCE node
I20260620 15:00:17.994618 5273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:17.996043 5273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:17.997239 5273 hybrid_clock.cc:648] HybridClock initialized: now 1781967617997200 us; error 41 us; skew 500 ppm
I20260620 15:00:17.997638 5273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:17 dist-test-slave-n5vv krb5kdc[5260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967617, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:18.000644 5273 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:00:18.002197 5273 webserver.cc:460] Webserver started at http://127.2.70.62:41067/ using document root <none> and password file <none>
I20260620 15:00:18.002872 5273 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:18.003008 5273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:18.003328 5273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:18.006342 5273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "0d289db278c5468981060d0f67d34210"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-n5vv"
server_key: "d11a0ab61ff64cbf95a04efc240eff50"
server_key_iv: "1f3a2a75466ec125622941a5cefd02aa"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.007126 5273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "0d289db278c5468981060d0f67d34210"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-n5vv"
server_key: "d11a0ab61ff64cbf95a04efc240eff50"
server_key_iv: "1f3a2a75466ec125622941a5cefd02aa"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.012486 5273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:00:18.015918 5288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:18.017242 5273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:18.017418 5273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "0d289db278c5468981060d0f67d34210"
format_stamp: "Formatted at 2026-06-20 15:00:18 on dist-test-slave-n5vv"
server_key: "d11a0ab61ff64cbf95a04efc240eff50"
server_key_iv: "1f3a2a75466ec125622941a5cefd02aa"
server_key_version: "encryptionkey@0"
I20260620 15:00:18.017669 5273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:18.036670 5273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:18.040972 5273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:18.041278 5273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:18.060428 5273 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:33891
I20260620 15:00:18.060446 5339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:33891 every 8 connection(s)
I20260620 15:00:18.061793 5273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:18.065333 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:18.069361 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 5273
I20260620 15:00:18.069626 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:18.070384 2328 external_mini_cluster.cc:1428] Setting key fb30209c35dc6695bf8a64d60e24d57a
I20260620 15:00:18.076834 5340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210: Bootstrap starting.
I20260620 15:00:18.080351 5340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:18.081475 5340 log.cc:824] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:18.084112 5340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210: No bootstrap required, opened a new log
Jun 20 15:00:18 dist-test-slave-n5vv krb5kdc[5260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:18.091805 5340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d289db278c5468981060d0f67d34210" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33891 } }
I20260620 15:00:18.092239 5340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:18.092363 5340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d289db278c5468981060d0f67d34210, State: Initialized, Role: FOLLOWER
I20260620 15:00:18.092948 5340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d289db278c5468981060d0f67d34210" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33891 } }
I20260620 15:00:18.093175 5340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:18.093304 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.072260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.2.70.62:33891)
0620 15:00:18.073030 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:18.073042 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:18.073081 (+ 39us) server_negotiation.cc:407] Connection header received
0620 15:00:18.075303 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:18.075326 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:18.075718 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:18.076032 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:18.077803 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.079132 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.080853 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.081580 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.081782 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.085801 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:18.085809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:18.085827 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:18.085869 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:18.089029 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.089771 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.089787 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.089793 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.089940 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:18.090329 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:18.090333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:18.090335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:18.090884 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:18.091190 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:18.091620 (+ 430us) server_negotiation.cc:299] Negotiation successful
0620 15:00:18.092005 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":377,"threads_started":1}
I20260620 15:00:18.093358 5340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:18.093583 5340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:18.096460 5340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d289db278c5468981060d0f67d34210" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33891 } }
I20260620 15:00:18.097029 5340 leader_election.cc:302] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d289db278c5468981060d0f67d34210; no voters:
I20260620 15:00:18.097612 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:00:18.097765 5340 leader_election.cc:288] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:18.097863 5345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:18.100303 5340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:18.101828 5345 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [term 1 LEADER]: Becoming Leader. State: Replica: 0d289db278c5468981060d0f67d34210, State: Running, Role: LEADER
I20260620 15:00:18.102628 5345 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d289db278c5468981060d0f67d34210" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33891 } }
I20260620 15:00:18.106055 5346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d289db278c5468981060d0f67d34210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d289db278c5468981060d0f67d34210" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33891 } } }
I20260620 15:00:18.106467 5346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:18.107035 5347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d289db278c5468981060d0f67d34210. Latest consensus state: current_term: 1 leader_uuid: "0d289db278c5468981060d0f67d34210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d289db278c5468981060d0f67d34210" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33891 } } }
I20260620 15:00:18.107434 5347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:18.110376 5352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:18.116405 5352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:18.134757 5352 catalog_manager.cc:1347] Generated new cluster ID: 2cd356f53260461fa8367c1c2e72763f
I20260620 15:00:18.134963 5352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:18.175540 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:18.166085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.2.70.62:33891)
0620 15:00:18.166267 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:18.166273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:18.166887 (+ 614us) server_negotiation.cc:407] Connection header received
0620 15:00:18.167044 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:18.167050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:18.167151 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:18.167280 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:18.168389 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.169685 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.171486 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:18.172273 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:18.172422 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:18.173665 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:18.173673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:18.173679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:18.174642 (+ 963us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:18.174730 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:18.174769 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:18.174905 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:18.175028 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 15:00:18.175143 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:00:18.191586 5352 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:18.193749 5352 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:18.203521 5352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d289db278c5468981060d0f67d34210: Generated new TSK 0
I20260620 15:00:18.204722 5352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:00:18.215862 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5273
I20260620 15:00:18.217079 5335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:18.311236 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 5273
2026-06-20T15:00:18Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1960 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:18 dist-test-slave-n5vv krb5kdc[5374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:18 dist-test-slave-n5vv krb5kdc[5374](info): set up 2 sockets
Jun 20 15:00:18 dist-test-slave-n5vv krb5kdc[5374](info): commencing operation
krb5kdc: starting...
W20260620 15:00:19.405849 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:19 dist-test-slave-n5vv krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:19.613603 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45159
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:45159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:19.935050 5387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:19.935328 5387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:19.935532 5387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:19.942665 5387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:19.942770 5387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:19.942885 5387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:19.943011 5387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:19.943074 5387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:19.953088 5387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:45159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:19.954494 5387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:19.955777 5387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:19.967470 5393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:19.967547 5392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:19.967515 5395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:19.968362 5387 server_base.cc:1034] running on GCE node
I20260620 15:00:19.969148 5387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:19.970542 5387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:19.971741 5387 hybrid_clock.cc:648] HybridClock initialized: now 1781967619971670 us; error 55 us; skew 500 ppm
I20260620 15:00:19.972079 5387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:19 dist-test-slave-n5vv krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967619, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:19.975483 5387 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:00:19.977006 5387 webserver.cc:460] Webserver started at http://127.2.70.62:35835/ using document root <none> and password file <none>
I20260620 15:00:19.977763 5387 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:19.977892 5387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:19.978199 5387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:19.981127 5387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "f54d1fe8d2d14801818af74a616d9bf7"
format_stamp: "Formatted at 2026-06-20 15:00:19 on dist-test-slave-n5vv"
server_key: "a76f4286e249d3d17046b3a1fe597ff3"
server_key_iv: "ddbcf661bf9a23a55ba363e50881a73d"
server_key_version: "encryptionkey@0"
I20260620 15:00:19.982120 5387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "f54d1fe8d2d14801818af74a616d9bf7"
format_stamp: "Formatted at 2026-06-20 15:00:19 on dist-test-slave-n5vv"
server_key: "a76f4286e249d3d17046b3a1fe597ff3"
server_key_iv: "ddbcf661bf9a23a55ba363e50881a73d"
server_key_version: "encryptionkey@0"
I20260620 15:00:19.987293 5387 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:19.990855 5402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:19.992223 5387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:19.992414 5387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "f54d1fe8d2d14801818af74a616d9bf7"
format_stamp: "Formatted at 2026-06-20 15:00:19 on dist-test-slave-n5vv"
server_key: "a76f4286e249d3d17046b3a1fe597ff3"
server_key_iv: "ddbcf661bf9a23a55ba363e50881a73d"
server_key_version: "encryptionkey@0"
I20260620 15:00:19.992611 5387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:20.014349 5387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:20.018613 5387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:20.018918 5387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:20.037892 5387 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:45159
I20260620 15:00:20.037916 5453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:45159 every 8 connection(s)
I20260620 15:00:20.039469 5387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:20.042037 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 5387
I20260620 15:00:20.042254 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:20.043152 2328 external_mini_cluster.cc:1428] Setting key 8d4568acc863f9fb5a6c998bd47355d9
I20260620 15:00:20.044243 5454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:00:20 dist-test-slave-n5vv krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:20.059271 5454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7: Bootstrap starting.
I20260620 15:00:20.062721 5454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:20.063760 5454 log.cc:824] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:20.066321 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:20.045357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.2.70.62:45159)
0620 15:00:20.046319 (+ 962us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:20.046331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:20.046376 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:00:20.048019 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:20.048041 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:20.048596 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:20.049017 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:20.050710 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:20.052167 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:20.053900 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:20.054550 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:20.054677 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:20.059218 (+ 4541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:20.059226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:20.059240 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:20.059282 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:20.062357 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:20.063012 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:20.063018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:20.063023 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:20.063181 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:20.063611 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:20.063615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:20.063617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:20.064155 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:20.064500 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:20.064876 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 15:00:20.065318 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":502,"threads_started":1}
I20260620 15:00:20.066794 5454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7: No bootstrap required, opened a new log
I20260620 15:00:20.074254 5454 raft_consensus.cc:348] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54d1fe8d2d14801818af74a616d9bf7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45159 } }
I20260620 15:00:20.074680 5454 raft_consensus.cc:374] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:20.074798 5454 raft_consensus.cc:729] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54d1fe8d2d14801818af74a616d9bf7, State: Initialized, Role: FOLLOWER
I20260620 15:00:20.075404 5454 consensus_queue.cc:260] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54d1fe8d2d14801818af74a616d9bf7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45159 } }
I20260620 15:00:20.075645 5454 raft_consensus.cc:388] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:20.075791 5454 raft_consensus.cc:482] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:20.075985 5454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:20.078442 5454 raft_consensus.cc:504] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54d1fe8d2d14801818af74a616d9bf7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45159 } }
I20260620 15:00:20.078972 5454 leader_election.cc:302] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f54d1fe8d2d14801818af74a616d9bf7; no voters:
I20260620 15:00:20.079617 5454 leader_election.cc:288] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:20.079766 5459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:20.080863 5459 raft_consensus.cc:686] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [term 1 LEADER]: Becoming Leader. State: Replica: f54d1fe8d2d14801818af74a616d9bf7, State: Running, Role: LEADER
I20260620 15:00:20.081413 5459 consensus_queue.cc:237] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54d1fe8d2d14801818af74a616d9bf7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45159 } }
I20260620 15:00:20.081897 5454 sys_catalog.cc:564] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:20.084550 5460 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f54d1fe8d2d14801818af74a616d9bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54d1fe8d2d14801818af74a616d9bf7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45159 } } }
I20260620 15:00:20.084522 5461 sys_catalog.cc:455] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f54d1fe8d2d14801818af74a616d9bf7. Latest consensus state: current_term: 1 leader_uuid: "f54d1fe8d2d14801818af74a616d9bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f54d1fe8d2d14801818af74a616d9bf7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45159 } } }
I20260620 15:00:20.084942 5460 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:20.084991 5461 sys_catalog.cc:458] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:20.086963 5464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:20.092538 5464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:20.102326 5464 catalog_manager.cc:1347] Generated new cluster ID: 4154e1ac2be248519ef611e16f79afe6
I20260620 15:00:20.102514 5464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:20.127471 5464 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:20.128829 5464 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:20.137857 5464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f54d1fe8d2d14801818af74a616d9bf7: Generated new TSK 0
I20260620 15:00:20.138962 5464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:00:20.197664 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45159
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:20.545845 5480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:20.546135 5480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:20.546286 5480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:20.553771 5480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:20.553891 5480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:20.554093 5480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:20.564277 5480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:20.565834 5480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:20.567304 5480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:20.580809 5485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:20.580840 5486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:20.580809 5488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:20.581705 5480 server_base.cc:1034] running on GCE node
I20260620 15:00:20.582340 5480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:20.583395 5480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:20.584630 5480 hybrid_clock.cc:648] HybridClock initialized: now 1781967620584566 us; error 64 us; skew 500 ppm
I20260620 15:00:20.584964 5480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:20 dist-test-slave-n5vv krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:20.588400 5480 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:00:20.590008 5480 webserver.cc:460] Webserver started at http://127.2.70.1:40383/ using document root <none> and password file <none>
I20260620 15:00:20.590727 5480 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:20.590870 5480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:20.591185 5480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:20.594193 5480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "a9d2882225214b5ba45e4828fdb58265"
format_stamp: "Formatted at 2026-06-20 15:00:20 on dist-test-slave-n5vv"
server_key: "ff460cebabe69742bbf83082e3c3d5ab"
server_key_iv: "c370e357b1332766c35c1bee7adf36ef"
server_key_version: "encryptionkey@0"
I20260620 15:00:20.594993 5480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "a9d2882225214b5ba45e4828fdb58265"
format_stamp: "Formatted at 2026-06-20 15:00:20 on dist-test-slave-n5vv"
server_key: "ff460cebabe69742bbf83082e3c3d5ab"
server_key_iv: "c370e357b1332766c35c1bee7adf36ef"
server_key_version: "encryptionkey@0"
I20260620 15:00:20.600298 5480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:00:20.603740 5495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:20.605049 5480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:00:20.605218 5480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "a9d2882225214b5ba45e4828fdb58265"
format_stamp: "Formatted at 2026-06-20 15:00:20 on dist-test-slave-n5vv"
server_key: "ff460cebabe69742bbf83082e3c3d5ab"
server_key_iv: "c370e357b1332766c35c1bee7adf36ef"
server_key_version: "encryptionkey@0"
I20260620 15:00:20.605419 5480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:20.639108 5480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:20.643163 5480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:20.643477 5480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:20.644414 5480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:20.645792 5480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:20.645900 5480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:20.646049 5480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:20.646124 5480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:20.678719 5480 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:41565
I20260620 15:00:20.678761 5607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:41565 every 8 connection(s)
I20260620 15:00:20.680006 5480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:20.689440 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 5480
I20260620 15:00:20.689714 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:20.690424 2328 external_mini_cluster.cc:1428] Setting key d56c26c181ccbd6891d21aa8c9e9ff81
Jun 20 15:00:20 dist-test-slave-n5vv krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967620, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:20.701260 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:20.682727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:49013 (local address 127.2.70.62:45159)
0620 15:00:20.683227 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:20.683233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:20.684166 (+ 933us) server_negotiation.cc:407] Connection header received
0620 15:00:20.685958 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:20.685962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:20.686029 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:20.686134 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:20.687630 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:20.688355 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:20.690156 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:20.690781 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:20.690927 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:20.695288 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:20.695293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:20.695297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:20.695327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:20.698147 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:20.699048 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:20.699052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:20.699055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:20.699165 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:20.699601 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:20.699607 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:20.699610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:20.699864 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:20.700015 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:20.700694 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 15:00:20.700899 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":257,"threads_started":1}
I20260620 15:00:20.702260 5611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:20.683342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45159 (local address 127.2.70.1:49013)
0620 15:00:20.684018 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:20.684055 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:20.685712 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:20.686335 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:20.686343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:20.686821 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:20.687492 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:20.687510 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:20.688534 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:20.688538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:20.690013 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:20.690024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:20.691032 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:20.691037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:20.691392 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:20.692096 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:20.692160 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:20.695042 (+ 2882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:20.698288 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:20.698300 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:20.698320 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:20.698938 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:20.699278 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:20.699284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:20.699287 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:20.699480 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:20.699992 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:20.700000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:20.700324 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 15:00:20.700587 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 15:00:20.701117 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":286,"threads_started":1}
I20260620 15:00:20.703845 5608 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45159
I20260620 15:00:20.704181 5608 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:20.705107 5608 heartbeater.cc:511] Master 127.2.70.62:45159 requested a full tablet report, sending...
I20260620 15:00:20.707661 5419 ts_manager.cc:194] Registered new tserver with Master: a9d2882225214b5ba45e4828fdb58265 (127.2.70.1:41565)
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:00:20.709838 5419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:49013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:00:20.733207 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45159
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:21.049620 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:21.049880 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:21.050074 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:21.057631 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:21.057736 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:21.057947 5614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:21.067588 5614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:21.068893 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:21.070200 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:21.083019 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:21.083048 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:21.083045 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:21.083956 5614 server_base.cc:1034] running on GCE node
I20260620 15:00:21.084649 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:21.085601 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:21.086802 5614 hybrid_clock.cc:648] HybridClock initialized: now 1781967621086754 us; error 37 us; skew 500 ppm
I20260620 15:00:21.087126 5614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:21.094867 5614 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:00:21.096458 5614 webserver.cc:460] Webserver started at http://127.2.70.2:44747/ using document root <none> and password file <none>
I20260620 15:00:21.097167 5614 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:21.097266 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:21.097690 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:21.100525 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "f332c13e71044c85b6a77a075518314d"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-n5vv"
server_key: "026fd28c959061fc4c2446d0d4ecefc1"
server_key_iv: "db79bb962cb48ff3d21e950c68daa7ef"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.101290 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "f332c13e71044c85b6a77a075518314d"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-n5vv"
server_key: "026fd28c959061fc4c2446d0d4ecefc1"
server_key_iv: "db79bb962cb48ff3d21e950c68daa7ef"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.106345 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 15:00:21.110134 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:21.111526 5614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:00:21.111657 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "f332c13e71044c85b6a77a075518314d"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-n5vv"
server_key: "026fd28c959061fc4c2446d0d4ecefc1"
server_key_iv: "db79bb962cb48ff3d21e950c68daa7ef"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.111856 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:21.127452 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:21.131618 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:21.131853 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:21.132720 5614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:21.134011 5614 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:21.134083 5614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:21.134160 5614 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:21.134207 5614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:21.167699 5614 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:39917
I20260620 15:00:21.167747 5741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:39917 every 8 connection(s)
I20260620 15:00:21.168979 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:21.170485 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 5614
I20260620 15:00:21.170678 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:21.171288 2328 external_mini_cluster.cc:1428] Setting key 2845f8a6bfba4bd6660e6cfafec6c5eb
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:00:21.191131 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.171688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:41743 (local address 127.2.70.62:45159)
0620 15:00:21.171852 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:21.171875 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:21.173307 (+ 1432us) server_negotiation.cc:407] Connection header received
0620 15:00:21.175172 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:21.175177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:21.175257 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:21.175378 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:21.176861 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.177653 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.179277 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.180088 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.180230 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.184091 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:21.184100 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:21.184108 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:21.184154 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:21.187840 (+ 3686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.188650 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.188657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.188661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.188812 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.189251 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.189259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.189264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.189675 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:21.189815 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:21.190593 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 15:00:21.190837 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:00:21.192219 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.172327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45159 (local address 127.2.70.2:41743)
0620 15:00:21.173147 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:21.173199 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:21.174917 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:21.175610 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:21.175618 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:21.176120 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:21.176698 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:21.176714 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.177847 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.177851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:21.179132 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:21.179144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.180308 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.180313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:21.180510 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.181229 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:21.181252 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:21.183845 (+ 2593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:21.187994 (+ 4149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:21.188001 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:21.188014 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:21.188519 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:21.188946 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:21.188950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:21.188952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:21.189133 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:21.189807 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:21.189813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:21.190201 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 15:00:21.190443 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 15:00:21.190859 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":338,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
I20260620 15:00:21.193888 5742 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45159
I20260620 15:00:21.194218 5742 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:21.194972 5742 heartbeater.cc:511] Master 127.2.70.62:45159 requested a full tablet report, sending...
I20260620 15:00:21.196596 5419 ts_manager.cc:194] Registered new tserver with Master: f332c13e71044c85b6a77a075518314d (127.2.70.2:39917)
I20260620 15:00:21.197870 5419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:41743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:00:21.216296 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45159
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:21.538903 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:21.539173 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:21.539330 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:21.546490 5747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:21.546586 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:21.546797 5747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:21.556432 5747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:21.557801 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:21.559079 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:21.572448 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:21.572520 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:21.572480 5753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:21.573319 5747 server_base.cc:1034] running on GCE node
I20260620 15:00:21.573890 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:21.574769 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:21.576005 5747 hybrid_clock.cc:648] HybridClock initialized: now 1781967621575952 us; error 45 us; skew 500 ppm
I20260620 15:00:21.576315 5747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:21.579365 5747 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:00:21.580734 5747 webserver.cc:460] Webserver started at http://127.2.70.3:43485/ using document root <none> and password file <none>
I20260620 15:00:21.581449 5747 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:21.581650 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:21.582005 5747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:21.584930 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "4b56b3a4ddd1453989809b9cc708fef8"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-n5vv"
server_key: "4eec53ecc46fab239b45ffddcab0137c"
server_key_iv: "5fc93ce1f1d3e777ec0ee26a2838cbb8"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.585779 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "4b56b3a4ddd1453989809b9cc708fef8"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-n5vv"
server_key: "4eec53ecc46fab239b45ffddcab0137c"
server_key_iv: "5fc93ce1f1d3e777ec0ee26a2838cbb8"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.590974 5747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:21.594446 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:21.596107 5747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:21.596343 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "4b56b3a4ddd1453989809b9cc708fef8"
format_stamp: "Formatted at 2026-06-20 15:00:21 on dist-test-slave-n5vv"
server_key: "4eec53ecc46fab239b45ffddcab0137c"
server_key_iv: "5fc93ce1f1d3e777ec0ee26a2838cbb8"
server_key_version: "encryptionkey@0"
I20260620 15:00:21.596622 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:21.627943 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:21.631937 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:21.632241 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:21.633226 5747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:21.634595 5747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:21.634697 5747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:21.634843 5747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:21.634922 5747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:21.667606 5747 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:40089
I20260620 15:00:21.667658 5874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:40089 every 8 connection(s)
I20260620 15:00:21.668871 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:21.673904 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 5747
I20260620 15:00:21.674099 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:21.674633 2328 external_mini_cluster.cc:1428] Setting key 64c679c6ee458109b16fd5f7e09a3956
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:21.689435 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.671472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:48255 (local address 127.2.70.62:45159)
0620 15:00:21.671644 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:21.671648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:21.672752 (+ 1104us) server_negotiation.cc:407] Connection header received
0620 15:00:21.675073 (+ 2321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:21.675077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:21.675191 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:21.675308 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:21.677198 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.678240 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.679773 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.680186 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.680321 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.683939 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:21.683945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:21.683949 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:21.683988 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:21.686469 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.687109 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.687113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.687116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.687244 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.687684 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.687687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.687689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.687947 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:21.688102 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:21.688867 (+ 765us) server_negotiation.cc:299] Negotiation successful
0620 15:00:21.689076 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:00:21.690492 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.671962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45159 (local address 127.2.70.3:48255)
0620 15:00:21.672610 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:21.672647 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:21.674770 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:21.675579 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:21.675591 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:21.676359 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:21.677045 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:21.677062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.678420 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.678426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:21.679622 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:21.679633 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.680405 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.680413 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:21.680661 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.681346 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:21.681368 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:21.683751 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:21.686597 (+ 2846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:21.686604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:21.686614 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:21.687001 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:21.687342 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:21.687345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:21.687348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:21.687534 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:21.688079 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:21.688087 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:21.688431 (+ 344us) client_negotiation.cc:769] Sending connection context
0620 15:00:21.688771 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 15:00:21.689280 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":270,"threads_started":1}
I20260620 15:00:21.691784 5875 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45159
I20260620 15:00:21.692054 5875 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:21.692695 5875 heartbeater.cc:511] Master 127.2.70.62:45159 requested a full tablet report, sending...
I20260620 15:00:21.694110 5419 ts_manager.cc:194] Registered new tserver with Master: 4b56b3a4ddd1453989809b9cc708fef8 (127.2.70.3:40089)
I20260620 15:00:21.694880 5419 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:48255
I20260620 15:00:21.702126 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:21.712805 5608 heartbeater.cc:503] Master 127.2.70.62:45159 was elected leader, sending a full tablet report...
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:21.749454 5610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.735688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49414 (local address 127.2.70.62:45159)
0620 15:00:21.735927 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:21.735935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:21.736246 (+ 311us) server_negotiation.cc:407] Connection header received
0620 15:00:21.736536 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:21.736540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:21.736606 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:21.736714 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:21.738129 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.738886 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.740153 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.740584 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.740714 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.744517 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:21.744522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:21.744525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:21.744561 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:21.747091 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.747668 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.747672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.747674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.747787 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.748107 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.748110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.748112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.748458 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:21.748632 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:21.748956 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 15:00:21.749114 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 15:00:21.750204 5419 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49414
I20260620 15:00:21.752748 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5480
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:00:21.768184 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.753846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39402 (local address 127.2.70.1:41565)
0620 15:00:21.754383 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:21.754391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:21.754413 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:00:21.754466 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:21.754472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:21.754698 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:21.754816 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:21.755953 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.756936 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.758188 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.758759 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.758859 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.762638 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:21.762649 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:21.762663 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:21.762693 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:21.765455 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.766385 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.766392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.766394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.766524 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.766844 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.766847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.766849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.767161 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:21.767380 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:21.767621 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 15:00:21.767843 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":266,"threads_started":1}
W20260620 15:00:21.769466 5603 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39402
W20260620 15:00:21.770687 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5480: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:21.770850 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 5480
I20260620 15:00:21.786401 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5614
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:00:21.801748 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.787273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45528 (local address 127.2.70.2:39917)
0620 15:00:21.787778 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:21.787785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:21.787807 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:00:21.787909 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:21.787918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:21.788138 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:21.788304 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:21.789571 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.790536 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.791792 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.792323 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.792479 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.796514 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:21.796524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:21.796537 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:21.796586 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:21.799248 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.799807 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.799814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.799816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.799972 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.800278 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.800282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.800284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.800654 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:21.800940 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:21.801177 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 15:00:21.801386 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":270,"threads_started":1}
W20260620 15:00:21.803009 5737 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45528
W20260620 15:00:21.804026 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5614: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:21.804195 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 5614
I20260620 15:00:21.819527 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5747
Jun 20 15:00:21 dist-test-slave-n5vv krb5kdc[5374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967621, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:00:21.835083 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:21.820408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49914 (local address 127.2.70.3:40089)
0620 15:00:21.821039 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:21.821046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:21.821076 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:00:21.821136 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:21.821142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:21.821367 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:21.821573 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:21.822774 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.823901 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.825178 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:21.825775 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:21.825875 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:21.830206 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:21.830217 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:21.830231 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:21.830262 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:21.832770 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.833346 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.833353 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.833355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.833458 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:21.833854 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:21.833857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:21.833859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:21.834163 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:21.834382 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:21.834629 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 15:00:21.834813 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":323,"threads_started":1}
W20260620 15:00:21.836308 5870 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49914
W20260620 15:00:21.837551 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5747: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:00:21.837705 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 5747
I20260620 15:00:21.853122 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5387
I20260620 15:00:21.854348 5449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:21.957358 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 5387
2026-06-20T15:00:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3651 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:22 dist-test-slave-n5vv krb5kdc[5892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:22 dist-test-slave-n5vv krb5kdc[5892](info): set up 2 sockets
Jun 20 15:00:22 dist-test-slave-n5vv krb5kdc[5892](info): commencing operation
krb5kdc: starting...
W20260620 15:00:23.053928 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:23 dist-test-slave-n5vv krb5kdc[5892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:23.250890 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:38043
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:38043
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:23.572142 5905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:23.572444 5905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:23.572618 5905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:23.579835 5905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:23.579946 5905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:23.580024 5905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:23.580106 5905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:23.580166 5905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:23.590049 5905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:38043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:38043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:23.591413 5905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:23.592823 5905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:23.604756 5910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.604806 5911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:23.604840 5913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:23.605780 5905 server_base.cc:1034] running on GCE node
I20260620 15:00:23.606645 5905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:23.608130 5905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:23.609355 5905 hybrid_clock.cc:648] HybridClock initialized: now 1781967623609311 us; error 32 us; skew 500 ppm
I20260620 15:00:23.609848 5905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:23 dist-test-slave-n5vv krb5kdc[5892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:23.613246 5905 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:00:23.614830 5905 webserver.cc:460] Webserver started at http://127.2.70.62:39575/ using document root <none> and password file <none>
I20260620 15:00:23.615557 5905 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:23.615689 5905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:23.616020 5905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:23.619136 5905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-n5vv"
server_key: "e87127ffe775be65f5015098c0c9d859"
server_key_iv: "795c328fc771c1f9437b5c4b3dea649f"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.619976 5905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-n5vv"
server_key: "e87127ffe775be65f5015098c0c9d859"
server_key_iv: "795c328fc771c1f9437b5c4b3dea649f"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.625185 5905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260620 15:00:23.628834 5920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:23.630391 5905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:23.630606 5905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb"
format_stamp: "Formatted at 2026-06-20 15:00:23 on dist-test-slave-n5vv"
server_key: "e87127ffe775be65f5015098c0c9d859"
server_key_iv: "795c328fc771c1f9437b5c4b3dea649f"
server_key_version: "encryptionkey@0"
I20260620 15:00:23.630820 5905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:23.648154 5905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:23.652762 5905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:23.653066 5905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:23.671644 5905 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:38043
I20260620 15:00:23.671685 5971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:38043 every 8 connection(s)
I20260620 15:00:23.673028 5905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:23.677168 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:23.679575 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 5905
I20260620 15:00:23.679777 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:23.680627 2328 external_mini_cluster.cc:1428] Setting key c25b0dd5cd5f944fdf2b7ab2eae3f273
I20260620 15:00:23.689750 5972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb: Bootstrap starting.
I20260620 15:00:23.693041 5972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:23.694142 5972 log.cc:824] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:00:23 dist-test-slave-n5vv krb5kdc[5892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:23.697212 5972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb: No bootstrap required, opened a new log
I20260620 15:00:23.704095 5975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:23.683132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.2.70.62:38043)
0620 15:00:23.683918 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:23.683931 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:23.683973 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:00:23.685965 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:23.685991 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:23.686498 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:23.686953 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:23.688410 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.689808 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.691117 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:23.691818 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:23.692018 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:23.696396 (+ 4378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:23.696442 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:23.696457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:23.696496 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:23.699757 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.700599 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.700609 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.700615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.700781 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:23.701173 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:23.701176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:23.701178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:23.701850 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:23.702063 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:23.702557 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 15:00:23.702976 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":333,"threads_started":1}
I20260620 15:00:23.706084 5972 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 38043 } }
I20260620 15:00:23.706461 5972 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:23.706588 5972 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0fdd28c167e46f7a44fa45ddebc8bdb, State: Initialized, Role: FOLLOWER
I20260620 15:00:23.707199 5972 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 38043 } }
I20260620 15:00:23.707442 5972 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:23.707588 5972 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:23.707769 5972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:23.710040 5972 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 38043 } }
I20260620 15:00:23.710532 5972 leader_election.cc:302] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0fdd28c167e46f7a44fa45ddebc8bdb; no voters:
I20260620 15:00:23.711089 5972 leader_election.cc:288] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:23.711225 5977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:23.711920 5977 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [term 1 LEADER]: Becoming Leader. State: Replica: b0fdd28c167e46f7a44fa45ddebc8bdb, State: Running, Role: LEADER
I20260620 15:00:23.712405 5977 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 38043 } }
I20260620 15:00:23.713173 5972 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:23.715529 5978 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 38043 } } }
I20260620 15:00:23.716056 5978 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:23.716157 5979 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0fdd28c167e46f7a44fa45ddebc8bdb. Latest consensus state: current_term: 1 leader_uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0fdd28c167e46f7a44fa45ddebc8bdb" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 38043 } } }
I20260620 15:00:23.716604 5979 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:23.718163 5984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:23.723641 5984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:23.732232 5984 catalog_manager.cc:1347] Generated new cluster ID: eefef56fd0324e0dad916269787fcb09
I20260620 15:00:23.732347 5984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:23.745038 5984 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:23.746443 5984 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:23.754038 5984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0fdd28c167e46f7a44fa45ddebc8bdb: Generated new TSK 0
I20260620 15:00:23.754922 5984 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:00:23.798575 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:38043
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:24.124626 5998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:24.124893 5998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:24.125051 5998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:24.132469 5998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:24.132572 5998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:24.132766 5998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:24.142593 5998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:38043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:24.143834 5998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:24.145184 5998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:24.158337 6004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.158486 6003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.158301 6006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:24.159293 5998 server_base.cc:1034] running on GCE node
I20260620 15:00:24.159849 5998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:24.160772 5998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:24.161988 5998 hybrid_clock.cc:648] HybridClock initialized: now 1781967624161930 us; error 48 us; skew 500 ppm
I20260620 15:00:24.162300 5998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:24 dist-test-slave-n5vv krb5kdc[5892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:24.165401 5998 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:00:24.166836 5998 webserver.cc:460] Webserver started at http://127.2.70.1:44517/ using document root <none> and password file <none>
I20260620 15:00:24.167503 5998 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:24.167646 5998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:24.167946 5998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:24.170827 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "de3488d1c7094a25a3bda6880f4fdfbd"
format_stamp: "Formatted at 2026-06-20 15:00:24 on dist-test-slave-n5vv"
server_key: "ef3ba7224f4ef0c4b0757c19acda3966"
server_key_iv: "648ac64ddb8b7ab97099f24a9761b326"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.171581 5998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "de3488d1c7094a25a3bda6880f4fdfbd"
format_stamp: "Formatted at 2026-06-20 15:00:24 on dist-test-slave-n5vv"
server_key: "ef3ba7224f4ef0c4b0757c19acda3966"
server_key_iv: "648ac64ddb8b7ab97099f24a9761b326"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.176478 5998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:00:24.180089 6013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.181428 5998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:24.181663 5998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "de3488d1c7094a25a3bda6880f4fdfbd"
format_stamp: "Formatted at 2026-06-20 15:00:24 on dist-test-slave-n5vv"
server_key: "ef3ba7224f4ef0c4b0757c19acda3966"
server_key_iv: "648ac64ddb8b7ab97099f24a9761b326"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.181869 5998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:24.208906 5998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:24.213451 5998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:24.213881 5998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:24.214989 5998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:24.216502 5998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:24.216614 5998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.216773 5998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:24.216869 5998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.252907 5998 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:39313
I20260620 15:00:24.252961 6125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:39313 every 8 connection(s)
I20260620 15:00:24.254493 5998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:24.257097 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 5998
I20260620 15:00:24.257530 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:24.258391 2328 external_mini_cluster.cc:1428] Setting key c5118d086564daee9a5f563386f0134c
Jun 20 15:00:24 dist-test-slave-n5vv krb5kdc[5892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:00:24.277217 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.257011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:49199 (local address 127.2.70.62:38043)
0620 15:00:24.257604 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.257608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.258750 (+ 1142us) server_negotiation.cc:407] Connection header received
0620 15:00:24.261239 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.261244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.261892 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.262024 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:24.263632 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.264418 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.266858 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.267289 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.267445 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.271361 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:24.271366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:24.271369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:24.271401 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:24.274178 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.274915 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.274921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.274924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.275046 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.275483 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.275487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.275490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.275747 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:24.275896 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.276645 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.276853 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":249,"threads_started":1}
I20260620 15:00:24.278373 6129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.257762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:38043 (local address 127.2.70.1:49199)
0620 15:00:24.258584 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.258625 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.260958 (+ 2333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.262287 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.262295 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.262847 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:24.263454 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.263473 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.264590 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.264596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.266693 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.266706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.267535 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.267545 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.267810 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.268533 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:24.268557 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:24.271127 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:24.274334 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:24.274344 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:24.274362 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:24.274783 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:24.275159 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:24.275164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:24.275168 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:24.275360 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:24.275879 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:24.275887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:24.276270 (+ 383us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.276534 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.277119 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":320,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
I20260620 15:00:24.280052 6126 heartbeater.cc:347] Connected to a master server at 127.2.70.62:38043
I20260620 15:00:24.280361 6126 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:24.281067 6126 heartbeater.cc:511] Master 127.2.70.62:38043 requested a full tablet report, sending...
I20260620 15:00:24.283185 5937 ts_manager.cc:194] Registered new tserver with Master: de3488d1c7094a25a3bda6880f4fdfbd (127.2.70.1:39313)
I20260620 15:00:24.284586 5937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:49199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:00:24.302884 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:38043
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:24.623644 6132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:24.623910 6132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:24.624009 6132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:24.631256 6132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:24.631354 6132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:24.631464 6132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:24.641571 6132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:38043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:24.642701 6132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:24.643910 6132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:24.656852 6140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.656819 6138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:24.656910 6137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:24.657769 6132 server_base.cc:1034] running on GCE node
I20260620 15:00:24.658366 6132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:24.659246 6132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:24.660490 6132 hybrid_clock.cc:648] HybridClock initialized: now 1781967624660435 us; error 47 us; skew 500 ppm
I20260620 15:00:24.660832 6132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:24 dist-test-slave-n5vv krb5kdc[5892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:24.664019 6132 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:00:24.665656 6132 webserver.cc:460] Webserver started at http://127.2.70.2:44571/ using document root <none> and password file <none>
I20260620 15:00:24.666337 6132 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:24.666483 6132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:24.666790 6132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:24.669737 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "58e257b8d2544af8a254f06c9a0aca9b"
format_stamp: "Formatted at 2026-06-20 15:00:24 on dist-test-slave-n5vv"
server_key: "7dfbc2a2384cf14ca84327ed948da0ef"
server_key_iv: "578436d7cfb30cbaa4dc46516e495e6d"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.670603 6132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "58e257b8d2544af8a254f06c9a0aca9b"
format_stamp: "Formatted at 2026-06-20 15:00:24 on dist-test-slave-n5vv"
server_key: "7dfbc2a2384cf14ca84327ed948da0ef"
server_key_iv: "578436d7cfb30cbaa4dc46516e495e6d"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.675798 6132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:24.679437 6147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.680903 6132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:24.681097 6132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "58e257b8d2544af8a254f06c9a0aca9b"
format_stamp: "Formatted at 2026-06-20 15:00:24 on dist-test-slave-n5vv"
server_key: "7dfbc2a2384cf14ca84327ed948da0ef"
server_key_iv: "578436d7cfb30cbaa4dc46516e495e6d"
server_key_version: "encryptionkey@0"
I20260620 15:00:24.681311 6132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:24.695847 6132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:24.700021 6132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:24.700309 6132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:24.701326 6132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:24.702651 6132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:24.702723 6132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.702891 6132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:24.702953 6132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:24.736313 6132 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:41651
I20260620 15:00:24.736356 6259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:41651 every 8 connection(s)
I20260620 15:00:24.737608 6132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:24.741667 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 6132
I20260620 15:00:24.741837 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:24.742380 2328 external_mini_cluster.cc:1428] Setting key 57d1e8881266db6682690dc7bea78ac5
Jun 20 15:00:24 dist-test-slave-n5vv krb5kdc[5892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967624, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:24.759670 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.740094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:47487 (local address 127.2.70.62:38043)
0620 15:00:24.740287 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:24.740294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:24.741712 (+ 1418us) server_negotiation.cc:407] Connection header received
0620 15:00:24.744001 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:24.744005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:24.744067 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:24.744183 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:24.745879 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.747089 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.748693 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.749432 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.749607 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.753625 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:24.753630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:24.753633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:24.753665 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:24.756244 (+ 2579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.757221 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.757225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.757227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.757357 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:24.757879 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:24.757882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:24.757884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:24.758156 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:24.758320 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:24.759097 (+ 777us) server_negotiation.cc:299] Negotiation successful
0620 15:00:24.759314 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:00:24.760982 6262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:24.740636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:38043 (local address 127.2.70.2:47487)
0620 15:00:24.741519 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:24.741580 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:24.743718 (+ 2138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:24.744509 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:24.744521 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:24.745071 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:24.745698 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.745716 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.747267 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.747272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.748546 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:24.748557 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:24.749678 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:24.749683 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:24.749874 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:24.750582 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:24.750606 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:24.753306 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:24.756389 (+ 3083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:24.756402 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:24.756423 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:24.757097 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:24.757469 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:24.757476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:24.757507 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:24.757771 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:24.758389 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:24.758396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:24.758720 (+ 324us) client_negotiation.cc:769] Sending connection context
0620 15:00:24.758981 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 15:00:24.759523 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":294,"threads_started":1}
I20260620 15:00:24.762436 6260 heartbeater.cc:347] Connected to a master server at 127.2.70.62:38043
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:00:24.762792 6260 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:24.763655 6260 heartbeater.cc:511] Master 127.2.70.62:38043 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
I20260620 15:00:24.765252 5937 ts_manager.cc:194] Registered new tserver with Master: 58e257b8d2544af8a254f06c9a0aca9b (127.2.70.2:41651)
I20260620 15:00:24.766104 5937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:47487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:00:24.787953 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:38043
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:35283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:25.107115 6265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:25.107453 6265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:25.107625 6265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:25.114899 6265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:25.115046 6265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:25.115211 6265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:25.126880 6265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:38043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:25.128293 6265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:25.129762 6265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:25.143550 6270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:25.143482 6271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:25.143515 6273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:25.146859 6265 server_base.cc:1034] running on GCE node
I20260620 15:00:25.147454 6265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:25.148368 6265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:25.149590 6265 hybrid_clock.cc:648] HybridClock initialized: now 1781967625149525 us; error 49 us; skew 500 ppm
I20260620 15:00:25.149902 6265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[5892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:25.152925 6265 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:00:25.154597 6265 webserver.cc:460] Webserver started at http://127.2.70.3:46005/ using document root <none> and password file <none>
I20260620 15:00:25.155288 6265 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:25.155429 6265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:25.155731 6265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:25.158859 6265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "34221d1a2cd14c60b3b066297d28bdfd"
format_stamp: "Formatted at 2026-06-20 15:00:25 on dist-test-slave-n5vv"
server_key: "e9ac8335b242b0144832903e181b3fc1"
server_key_iv: "a8bf9b9656772256784982dcfa9125d0"
server_key_version: "encryptionkey@0"
I20260620 15:00:25.159641 6265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "34221d1a2cd14c60b3b066297d28bdfd"
format_stamp: "Formatted at 2026-06-20 15:00:25 on dist-test-slave-n5vv"
server_key: "e9ac8335b242b0144832903e181b3fc1"
server_key_iv: "a8bf9b9656772256784982dcfa9125d0"
server_key_version: "encryptionkey@0"
I20260620 15:00:25.164937 6265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:25.168498 6280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:25.169965 6265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:25.170135 6265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "34221d1a2cd14c60b3b066297d28bdfd"
format_stamp: "Formatted at 2026-06-20 15:00:25 on dist-test-slave-n5vv"
server_key: "e9ac8335b242b0144832903e181b3fc1"
server_key_iv: "a8bf9b9656772256784982dcfa9125d0"
server_key_version: "encryptionkey@0"
I20260620 15:00:25.170346 6265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:25.206590 6265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:25.210711 6265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:25.210999 6265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:25.211980 6265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:25.213295 6265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:25.213399 6265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:25.213594 6265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:25.213663 6265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:25.247314 6265 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:41521
I20260620 15:00:25.247363 6392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:41521 every 8 connection(s)
I20260620 15:00:25.248601 6265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:25.256188 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 6265
I20260620 15:00:25.256352 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:25.256873 2328 external_mini_cluster.cc:1428] Setting key c386a91f98689a3e6218ba14323115eb
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[5892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:25.268029 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.251186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:41547 (local address 127.2.70.62:38043)
0620 15:00:25.251393 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.251400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.252628 (+ 1228us) server_negotiation.cc:407] Connection header received
0620 15:00:25.254596 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.254601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.254697 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.254809 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:25.256318 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.257282 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.259028 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.259439 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.259596 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.263070 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:25.263075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:25.263078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:25.263110 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:25.265540 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.266207 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.266210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.266213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.266329 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.266687 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.266690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.266692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.266922 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:25.267020 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.267645 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.267801 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:00:25.269071 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.251685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:38043 (local address 127.2.70.3:41547)
0620 15:00:25.252456 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:25.252506 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:25.254315 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:25.255057 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:25.255065 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:25.255567 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:25.256144 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.256161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.257435 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.257447 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.258896 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:25.258912 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.259718 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.259726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:25.260053 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.260675 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:25.260701 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:25.262863 (+ 2162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:25.265668 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:25.265675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:25.265688 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:25.266102 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:25.266418 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:25.266421 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:25.266423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:25.266593 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:25.267019 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:25.267025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:25.267313 (+ 288us) client_negotiation.cc:769] Sending connection context
0620 15:00:25.267539 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 15:00:25.267897 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":303,"threads_started":1}
I20260620 15:00:25.270562 6393 heartbeater.cc:347] Connected to a master server at 127.2.70.62:38043
I20260620 15:00:25.270879 6393 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:25.271677 6393 heartbeater.cc:511] Master 127.2.70.62:38043 requested a full tablet report, sending...
I20260620 15:00:25.273066 5937 ts_manager.cc:194] Registered new tserver with Master: 34221d1a2cd14c60b3b066297d28bdfd (127.2.70.3:41521)
I20260620 15:00:25.274000 5937 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:41547
I20260620 15:00:25.284186 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:25.287717 6126 heartbeater.cc:503] Master 127.2.70.62:38043 was elected leader, sending a full tablet report...
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[5892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[5892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:00:25.331112 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.317038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.2.70.1:39313)
0620 15:00:25.317690 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.317697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.317726 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:00:25.317812 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.317818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.318036 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.318170 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:25.319447 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.320430 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.321689 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.322261 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.322378 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.326110 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:25.326120 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:25.326134 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:25.326166 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:25.328758 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.329353 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.329360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.329362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.329470 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.329845 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.329848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.329850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.330164 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:25.330375 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.330626 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.330803 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":288,"threads_started":1}
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[5892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:25.348344 6128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.335288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.2.70.62:38043)
0620 15:00:25.335545 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.335554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.335787 (+ 233us) server_negotiation.cc:407] Connection header received
0620 15:00:25.336000 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.336004 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.336061 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.336160 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:25.337602 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.338289 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.339484 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.339865 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.339955 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.343718 (+ 3763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:25.343722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:25.343726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:25.343757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:25.346166 (+ 2409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.346728 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.346731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.346733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.346835 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.347124 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.347127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.347129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.347399 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:25.347569 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.347850 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.348019 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 15:00:25.348879 5937 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54372
I20260620 15:00:25.349897 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5998
I20260620 15:00:25.362401 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.350820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44294 (local address 127.2.70.1:39313)
0620 15:00:25.351040 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.351047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.351251 (+ 204us) server_negotiation.cc:407] Connection header received
0620 15:00:25.351444 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.351448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.351534 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.351636 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:25.352756 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.353417 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.354850 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.355251 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.355383 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.357534 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:25.357543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:25.357548 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:25.357589 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:25.359845 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.360500 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.360505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.360509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.360632 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.360982 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.360996 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.360999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.361241 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:25.361540 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.361830 (+ 290us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.362072 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:00:25.363049 6121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:25.445879 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 5998
I20260620 15:00:25.462316 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 6132
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[5892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:00:25.477761 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.463223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.2.70.2:41651)
0620 15:00:25.463752 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.463759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.463785 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:00:25.463844 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.463851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.464094 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.464226 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:25.465285 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.466352 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.467565 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.468098 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.468191 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.472359 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:25.472370 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:25.472385 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:25.472419 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:25.475354 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.475966 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.475972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.475975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.476083 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.476372 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.476375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.476377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.476695 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:25.476937 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.477153 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.477333 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":287,"threads_started":1}
I20260620 15:00:25.479130 6255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:25.561168 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 6132
I20260620 15:00:25.577354 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 6265
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[5892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:00:25.593101 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:25.578317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.2.70.3:41521)
0620 15:00:25.578849 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:25.578856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:25.578883 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:00:25.578945 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:25.578951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:25.579166 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:25.579295 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:25.580456 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.581466 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.582884 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:25.583435 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:25.583531 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:25.587854 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:25.587865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:25.587879 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:25.587912 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:25.590795 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.591389 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.591395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.591398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.591511 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:25.591798 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:25.591801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:25.591803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:25.592125 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:25.592338 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:25.592629 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:00:25.592811 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":234,"threads_started":1}
I20260620 15:00:25.594376 6388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:25.677610 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 6265
I20260620 15:00:25.689976 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 5905
I20260620 15:00:25.691181 5967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:25.792279 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 5905
2026-06-20T15:00:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3834 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[6412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[6412](info): set up 2 sockets
Jun 20 15:00:25 dist-test-slave-n5vv krb5kdc[6412](info): commencing operation
krb5kdc: starting...
W20260620 15:00:26.890019 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:27 dist-test-slave-n5vv krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:27.097667 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:36389
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:36389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:27.418241 6425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:27.418592 6425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:27.418762 6425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:27.426148 6425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:27.426302 6425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:27.426414 6425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:27.426529 6425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:27.426612 6425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:27.436667 6425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:36389
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:36389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:27.438222 6425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:27.439568 6425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:27.451138 6433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:27.451254 6431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:27.451231 6430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:27.452174 6425 server_base.cc:1034] running on GCE node
I20260620 15:00:27.453022 6425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:27.454440 6425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:27.455710 6425 hybrid_clock.cc:648] HybridClock initialized: now 1781967627455657 us; error 46 us; skew 500 ppm
I20260620 15:00:27.456108 6425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:27 dist-test-slave-n5vv krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967627, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:27.464437 6425 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:00:27.465792 6425 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:27.465936 6425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:27.466269 6425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:27.469218 6425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b"
format_stamp: "Formatted at 2026-06-20 15:00:27 on dist-test-slave-n5vv"
server_key: "8667b4fee29fa1744288ea45282a5280"
server_key_iv: "f059c0d4b57cf47f9baa02ab07dcc3db"
server_key_version: "encryptionkey@0"
I20260620 15:00:27.470094 6425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b"
format_stamp: "Formatted at 2026-06-20 15:00:27 on dist-test-slave-n5vv"
server_key: "8667b4fee29fa1744288ea45282a5280"
server_key_iv: "f059c0d4b57cf47f9baa02ab07dcc3db"
server_key_version: "encryptionkey@0"
I20260620 15:00:27.475102 6425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:27.478479 6439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:27.479796 6425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:27.480002 6425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b"
format_stamp: "Formatted at 2026-06-20 15:00:27 on dist-test-slave-n5vv"
server_key: "8667b4fee29fa1744288ea45282a5280"
server_key_iv: "f059c0d4b57cf47f9baa02ab07dcc3db"
server_key_version: "encryptionkey@0"
I20260620 15:00:27.480248 6425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:27.516000 6425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:27.520085 6425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:27.520359 6425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:27.538679 6425 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:36389
I20260620 15:00:27.538703 6490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:36389 every 8 connection(s)
I20260620 15:00:27.539601 6425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:27.543220 6491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:27.547729 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 6425
I20260620 15:00:27.547950 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:27.548743 2328 external_mini_cluster.cc:1428] Setting key ac4d9ed4c8b58b5e68a2c06f020078aa
I20260620 15:00:27.554869 6491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b: Bootstrap starting.
I20260620 15:00:27.558452 6491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:27.559635 6491 log.cc:824] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:27.563109 6491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b: No bootstrap required, opened a new log
Jun 20 15:00:27 dist-test-slave-n5vv krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:27.570820 6491 raft_consensus.cc:348] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36389 } }
I20260620 15:00:27.571308 6491 raft_consensus.cc:374] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:27.571431 6491 raft_consensus.cc:729] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec8e6a8d77a4c5fa82c3f7f1f589d7b, State: Initialized, Role: FOLLOWER
I20260620 15:00:27.572080 6491 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36389 } }
I20260620 15:00:27.572316 6491 raft_consensus.cc:388] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:27.572402 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:27.550810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.2.70.62:36389)
0620 15:00:27.551627 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:27.551639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:27.551685 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:00:27.553386 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:27.553410 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:27.553867 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:27.554177 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:27.555995 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.557324 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.559204 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:27.560018 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:27.560228 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:27.564818 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:27.564829 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:27.564898 (+ 69us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:27.564948 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:27.568108 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:27.568808 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:27.568820 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:27.568826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:27.568993 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:27.569389 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:27.569394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:27.569397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:27.570034 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:27.570348 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:27.570780 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 15:00:27.571236 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":405,"threads_started":1}
I20260620 15:00:27.572460 6491 raft_consensus.cc:482] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:27.572662 6491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:27.575784 6491 raft_consensus.cc:504] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36389 } }
I20260620 15:00:27.576287 6491 leader_election.cc:302] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec8e6a8d77a4c5fa82c3f7f1f589d7b; no voters:
I20260620 15:00:27.576836 6491 leader_election.cc:288] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:27.576967 6496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:27.577682 6496 raft_consensus.cc:686] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [term 1 LEADER]: Becoming Leader. State: Replica: cec8e6a8d77a4c5fa82c3f7f1f589d7b, State: Running, Role: LEADER
I20260620 15:00:27.578266 6496 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36389 } }
I20260620 15:00:27.578868 6491 sys_catalog.cc:564] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:27.580785 6498 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec8e6a8d77a4c5fa82c3f7f1f589d7b. Latest consensus state: current_term: 1 leader_uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36389 } } }
I20260620 15:00:27.581126 6498 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:27.581753 6497 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec8e6a8d77a4c5fa82c3f7f1f589d7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36389 } } }
I20260620 15:00:27.582039 6497 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:27.582093 6501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:27.588186 6501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:27.597354 6501 catalog_manager.cc:1347] Generated new cluster ID: bda52a5f3bbb40c1b25eb4e2bcf74fda
I20260620 15:00:27.597471 6501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:27.616458 6501 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:27.617849 6501 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:27.624701 6501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b: Generated new TSK 0
I20260620 15:00:27.625577 6501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:00:27.681254 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36389
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:28.022331 6517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:28.022624 6517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:28.022723 6517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:28.030103 6517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:28.030216 6517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:28.030333 6517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:28.040452 6517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:28.041890 6517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:28.043125 6517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:28.055830 6523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:28.055946 6522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:28.055963 6525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:28.056594 6517 server_base.cc:1034] running on GCE node
I20260620 15:00:28.057229 6517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:28.058261 6517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:28.059455 6517 hybrid_clock.cc:648] HybridClock initialized: now 1781967628059420 us; error 30 us; skew 500 ppm
I20260620 15:00:28.059784 6517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:28 dist-test-slave-n5vv krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:28.063050 6517 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:00:28.064159 6517 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:28.064283 6517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:28.064616 6517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:28.067706 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "467d39b0935e40e4ae5e52be84ebc572"
format_stamp: "Formatted at 2026-06-20 15:00:28 on dist-test-slave-n5vv"
server_key: "29c438293bba0203b7b29e3e95aca0ae"
server_key_iv: "3c0b1d2a736a86e0c1da98fe448ea801"
server_key_version: "encryptionkey@0"
I20260620 15:00:28.068521 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "467d39b0935e40e4ae5e52be84ebc572"
format_stamp: "Formatted at 2026-06-20 15:00:28 on dist-test-slave-n5vv"
server_key: "29c438293bba0203b7b29e3e95aca0ae"
server_key_iv: "3c0b1d2a736a86e0c1da98fe448ea801"
server_key_version: "encryptionkey@0"
I20260620 15:00:28.073599 6517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:00:28.076833 6531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:28.078123 6517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:28.078293 6517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "467d39b0935e40e4ae5e52be84ebc572"
format_stamp: "Formatted at 2026-06-20 15:00:28 on dist-test-slave-n5vv"
server_key: "29c438293bba0203b7b29e3e95aca0ae"
server_key_iv: "3c0b1d2a736a86e0c1da98fe448ea801"
server_key_version: "encryptionkey@0"
I20260620 15:00:28.078521 6517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:28.106846 6517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:28.111338 6517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:28.111630 6517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:28.112608 6517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:28.114322 6517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:28.114444 6517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:28.114616 6517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:28.114717 6517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:28.146956 6517 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:45989
I20260620 15:00:28.147010 6643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:45989 every 8 connection(s)
I20260620 15:00:28.147816 6517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:28.153949 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 6517
I20260620 15:00:28.154147 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:28.154768 2328 external_mini_cluster.cc:1428] Setting key 03ee1203119028299d98b414bf868a84
Jun 20 15:00:28 dist-test-slave-n5vv krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:28.169945 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:28.150222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:40185 (local address 127.2.70.62:36389)
0620 15:00:28.150666 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:28.150672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:28.151649 (+ 977us) server_negotiation.cc:407] Connection header received
0620 15:00:28.153399 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:28.153404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:28.153478 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:28.153635 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:28.155310 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:28.156172 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:28.157960 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:28.158721 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:28.158868 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:28.162815 (+ 3947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:28.162820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:28.162823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:28.162853 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:28.166382 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:28.167238 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:28.167246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:28.167253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:28.167402 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:28.167891 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:28.167898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:28.167904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:28.168267 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:28.168423 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:28.169143 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 15:00:28.169401 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":213,"threads_started":1}
I20260620 15:00:28.170724 6647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:28.150775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:36389 (local address 127.2.70.1:40185)
0620 15:00:28.151487 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:28.151525 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:28.153151 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:28.153899 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:28.153907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:28.154530 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:28.155141 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:28.155159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:28.156421 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:28.156427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:28.157814 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:28.157826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:28.158988 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:28.158998 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:28.159352 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:28.160098 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:28.160124 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:28.162594 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:28.166596 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:28.166604 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:28.166618 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:28.167108 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:28.167551 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:28.167555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:28.167557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:28.167772 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:28.168411 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:28.168417 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:28.168748 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 15:00:28.169020 (+ 272us) client_negotiation.cc:240] Negotiation successful
0620 15:00:28.169421 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":280,"threads_started":1}
I20260620 15:00:28.172557 6644 heartbeater.cc:347] Connected to a master server at 127.2.70.62:36389
I20260620 15:00:28.172884 6644 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:28.173605 6644 heartbeater.cc:511] Master 127.2.70.62:36389 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:00:28.175678 6456 ts_manager.cc:194] Registered new tserver with Master: 467d39b0935e40e4ae5e52be84ebc572 (127.2.70.1:45989)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
I20260620 15:00:28.177194 6456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:40185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:00:28.200721 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36389
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:28.525038 6650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:28.525297 6650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:28.525403 6650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:28.532639 6650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:28.532780 6650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:28.532934 6650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:28.542572 6650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:28.543808 6650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:28.545104 6650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:28.558108 6655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:28.558331 6658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:28.558406 6656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:28.559294 6650 server_base.cc:1034] running on GCE node
I20260620 15:00:28.559841 6650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:28.560714 6650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:28.561942 6650 hybrid_clock.cc:648] HybridClock initialized: now 1781967628561904 us; error 52 us; skew 500 ppm
I20260620 15:00:28.562245 6650 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:28 dist-test-slave-n5vv krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:28.565140 6650 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:00:28.566367 6650 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:28.566500 6650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:28.566828 6650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:28.569804 6650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "1a62656f90ef45fdbd14377269fb5456"
format_stamp: "Formatted at 2026-06-20 15:00:28 on dist-test-slave-n5vv"
server_key: "a56a562abb125fe41ba805b05f5f40c9"
server_key_iv: "c8c0b8b9e558fc489904bcff1b0baaf2"
server_key_version: "encryptionkey@0"
I20260620 15:00:28.570609 6650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "1a62656f90ef45fdbd14377269fb5456"
format_stamp: "Formatted at 2026-06-20 15:00:28 on dist-test-slave-n5vv"
server_key: "a56a562abb125fe41ba805b05f5f40c9"
server_key_iv: "c8c0b8b9e558fc489904bcff1b0baaf2"
server_key_version: "encryptionkey@0"
I20260620 15:00:28.575601 6650 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:28.579155 6664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:28.580540 6650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:00:28.580716 6650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "1a62656f90ef45fdbd14377269fb5456"
format_stamp: "Formatted at 2026-06-20 15:00:28 on dist-test-slave-n5vv"
server_key: "a56a562abb125fe41ba805b05f5f40c9"
server_key_iv: "c8c0b8b9e558fc489904bcff1b0baaf2"
server_key_version: "encryptionkey@0"
I20260620 15:00:28.580948 6650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:28.607831 6650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:28.611894 6650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:28.612174 6650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:28.613134 6650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:28.614444 6650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:28.614565 6650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:28.614698 6650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:28.614785 6650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:28.647807 6650 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:37217
I20260620 15:00:28.647846 6776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:37217 every 8 connection(s)
I20260620 15:00:28.648671 6650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:28.649875 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 6650
I20260620 15:00:28.650050 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:28.650681 2328 external_mini_cluster.cc:1428] Setting key 8f407c00913875ce31822f9a75756ae3
Jun 20 15:00:28 dist-test-slave-n5vv krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967628, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:00:28.670248 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:28.651493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:43371 (local address 127.2.70.62:36389)
0620 15:00:28.651688 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:28.651693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:28.653117 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 15:00:28.654993 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:28.655001 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:28.655092 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:28.655205 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:28.656664 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:28.657891 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:28.659903 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:28.660340 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:28.660536 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:28.664088 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:28.664093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:28.664097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:28.664128 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:28.666790 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:28.667838 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:28.667845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:28.667849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:28.667990 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:28.668442 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:28.668448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:28.668451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:28.668732 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:28.668890 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:28.669666 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 15:00:28.669895 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:00:28.671447 6779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:28.652077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:36389 (local address 127.2.70.2:43371)
0620 15:00:28.652955 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:28.653000 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:28.654720 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:28.655398 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:28.655405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:28.655930 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:28.656486 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:28.656503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:28.658086 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:28.658093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:28.659752 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:28.659764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:28.660633 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:28.660640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:28.660861 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:28.661664 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:28.661686 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:28.663823 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:28.666934 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:28.666945 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:28.666962 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:28.667550 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:28.668120 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:28.668125 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:28.668129 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:28.668322 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:28.668868 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:28.668876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:28.669237 (+ 361us) client_negotiation.cc:769] Sending connection context
0620 15:00:28.669570 (+ 333us) client_negotiation.cc:240] Negotiation successful
0620 15:00:28.670124 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":433,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
I20260620 15:00:28.673058 6777 heartbeater.cc:347] Connected to a master server at 127.2.70.62:36389
I20260620 15:00:28.673362 6777 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:28.674036 6777 heartbeater.cc:511] Master 127.2.70.62:36389 requested a full tablet report, sending...
I20260620 15:00:28.675415 6456 ts_manager.cc:194] Registered new tserver with Master: 1a62656f90ef45fdbd14377269fb5456 (127.2.70.2:37217)
I20260620 15:00:28.676213 6456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:43371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:00:28.696096 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36389
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:29.015823 6782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:29.016122 6782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:29.016220 6782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:29.023504 6782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:29.023617 6782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:29.023736 6782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:29.033972 6782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36389
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:29.035198 6782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:29.036430 6782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:29.048995 6787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:29.049031 6790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:29.049074 6788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:29.049810 6782 server_base.cc:1034] running on GCE node
I20260620 15:00:29.050463 6782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:29.051435 6782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:29.052675 6782 hybrid_clock.cc:648] HybridClock initialized: now 1781967629052612 us; error 59 us; skew 500 ppm
I20260620 15:00:29.053009 6782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:29 dist-test-slave-n5vv krb5kdc[6412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:29.056046 6782 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:00:29.057130 6782 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:29.057241 6782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:29.057673 6782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:29.060463 6782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "c319b528070b4e56a9dc6846f3e2db82"
format_stamp: "Formatted at 2026-06-20 15:00:29 on dist-test-slave-n5vv"
server_key: "a913353a0192364d20729379c80b65c7"
server_key_iv: "ff0cb4aa45666a8f2cff1b855318ee2f"
server_key_version: "encryptionkey@0"
I20260620 15:00:29.061260 6782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "c319b528070b4e56a9dc6846f3e2db82"
format_stamp: "Formatted at 2026-06-20 15:00:29 on dist-test-slave-n5vv"
server_key: "a913353a0192364d20729379c80b65c7"
server_key_iv: "ff0cb4aa45666a8f2cff1b855318ee2f"
server_key_version: "encryptionkey@0"
I20260620 15:00:29.066242 6782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 15:00:29.069785 6796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:29.071082 6782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:00:29.071250 6782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "c319b528070b4e56a9dc6846f3e2db82"
format_stamp: "Formatted at 2026-06-20 15:00:29 on dist-test-slave-n5vv"
server_key: "a913353a0192364d20729379c80b65c7"
server_key_iv: "ff0cb4aa45666a8f2cff1b855318ee2f"
server_key_version: "encryptionkey@0"
I20260620 15:00:29.071467 6782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:29.103765 6782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:29.107685 6782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:29.107961 6782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:29.108907 6782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:29.110212 6782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:29.110296 6782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:29.110455 6782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:29.110548 6782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:29.142427 6782 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:42273
I20260620 15:00:29.142472 6908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:42273 every 8 connection(s)
I20260620 15:00:29.143292 6782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:29.144743 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 6782
I20260620 15:00:29.144920 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:29.145382 2328 external_mini_cluster.cc:1428] Setting key 83391f102bb81c670a58b953e2214fed
Jun 20 15:00:29 dist-test-slave-n5vv krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967629, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:29.162931 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.146045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:47409 (local address 127.2.70.62:36389)
0620 15:00:29.146234 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.146241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.147631 (+ 1390us) server_negotiation.cc:407] Connection header received
0620 15:00:29.149521 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.149526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.149612 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.149717 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:29.151162 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.152265 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.153908 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.154305 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.154402 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.157660 (+ 3258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:29.157665 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:29.157668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:29.157698 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:29.160242 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.161024 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.161028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.161030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.161148 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.161541 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.161549 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.161551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.161784 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:29.161894 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.162542 (+ 648us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.162691 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:00:29.163851 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.146601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:36389 (local address 127.2.70.3:47409)
0620 15:00:29.147475 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.147528 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:29.149217 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:29.149929 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:29.149936 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:29.150442 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:29.150972 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.150999 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.152410 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.152414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.153785 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.153797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.154488 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.154495 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.154672 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.155334 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:29.155356 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:29.157395 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:29.160346 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:29.160355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:29.160369 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:29.160938 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:29.161239 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:29.161242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:29.161244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:29.161407 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:29.161893 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:29.161898 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:29.162217 (+ 319us) client_negotiation.cc:769] Sending connection context
0620 15:00:29.162451 (+ 234us) client_negotiation.cc:240] Negotiation successful
0620 15:00:29.162806 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":391,"threads_started":1}
I20260620 15:00:29.165179 6909 heartbeater.cc:347] Connected to a master server at 127.2.70.62:36389
I20260620 15:00:29.165477 6909 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:29.166157 6909 heartbeater.cc:511] Master 127.2.70.62:36389 requested a full tablet report, sending...
I20260620 15:00:29.167521 6456 ts_manager.cc:194] Registered new tserver with Master: c319b528070b4e56a9dc6846f3e2db82 (127.2.70.3:42273)
I20260620 15:00:29.168232 6456 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:47409
I20260620 15:00:29.172680 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:29.180130 6644 heartbeater.cc:503] Master 127.2.70.62:36389 was elected leader, sending a full tablet report...
I20260620 15:00:29.190081 6646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.177313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34590 (local address 127.2.70.62:36389)
0620 15:00:29.177531 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.177541 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.177972 (+ 431us) server_negotiation.cc:407] Connection header received
0620 15:00:29.178168 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.178177 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.178233 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.178334 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:29.179804 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.180745 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.182100 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.182520 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.182672 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.185021 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:29.185027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:29.185038 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:29.185091 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:29.187562 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.188172 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.188177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.188181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.188296 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.188648 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.188654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.188658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.188947 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:29.189223 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.189546 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.189762 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:00:29.194476 6456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:29.201224 6456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:29.222874 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.213390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.2.70.3:42273)
0620 15:00:29.214017 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.214026 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.214060 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:00:29.214355 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.214385 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.214650 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.214876 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:29.216019 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.217673 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.220346 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.221697 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.221824 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.222064 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.222339 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.222525 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":488,"threads_started":1}
I20260620 15:00:29.222929 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.213018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:42273 (local address 127.0.0.1:46948)
0620 15:00:29.213910 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.213950 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:29.214131 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:29.215297 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:29.215304 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:29.215333 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:29.215770 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.215783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.217807 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.217813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.220167 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.220182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.221893 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.221900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.222092 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.222143 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 15:00:29.222283 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 15:00:29.222462 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":37,"thread_start_us":454,"threads_started":1}
I20260620 15:00:29.223387 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.214343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:37217 (local address 127.0.0.1:37764)
0620 15:00:29.214950 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.214968 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:29.215068 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:29.216706 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:29.216710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:29.216732 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:29.217128 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.217144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.218713 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.218721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.220736 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.220749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.222521 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.222527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.222692 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.222732 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:00:29.222798 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 15:00:29.222945 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":1,"thread_start_us":263,"threads_started":1}
I20260620 15:00:29.223824 6925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.215048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:45989 (local address 127.0.0.1:58678)
0620 15:00:29.215891 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.215907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:29.216003 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:29.217060 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:29.217067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:29.217096 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:29.217430 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.217438 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.219367 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.219373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.221290 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.221300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.223066 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.223073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.223258 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.223296 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:00:29.223364 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 15:00:29.223478 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":54,"thread_start_us":732,"threads_started":1}
I20260620 15:00:29.224155 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.214608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.2.70.2:37217)
0620 15:00:29.215549 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.215558 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.215586 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:00:29.215678 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.215686 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.216499 (+ 813us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.217284 (+ 785us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:29.217541 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.218584 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.220894 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.222291 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.223417 (+ 1126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.223625 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.223686 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.223863 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":658,"threads_started":1}
I20260620 15:00:29.224838 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.215606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.2.70.1:45989)
0620 15:00:29.216561 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.216570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.216595 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:00:29.216682 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.216689 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.216890 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.217063 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:29.217594 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.219218 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.221439 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.222817 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.224036 (+ 1219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.224259 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.224343 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.224521 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":616,"threads_started":1}
I20260620 15:00:29.226956 6844 tablet_service.cc:1467] Processing CreateTablet for tablet 8d315f8022414f50b13e613e76842a28 (DEFAULT_TABLE table=test-table [id=02521812c3f04a1c90ce28e6aaf6067e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:29.228153 6844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d315f8022414f50b13e613e76842a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:29.227619 6712 tablet_service.cc:1467] Processing CreateTablet for tablet 8d315f8022414f50b13e613e76842a28 (DEFAULT_TABLE table=test-table [id=02521812c3f04a1c90ce28e6aaf6067e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:29.228329 6579 tablet_service.cc:1467] Processing CreateTablet for tablet 8d315f8022414f50b13e613e76842a28 (DEFAULT_TABLE table=test-table [id=02521812c3f04a1c90ce28e6aaf6067e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:29.228932 6712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d315f8022414f50b13e613e76842a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:29.229660 6579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d315f8022414f50b13e613e76842a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:29.239020 6927 tablet_bootstrap.cc:492] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: Bootstrap starting.
I20260620 15:00:29.243295 6928 tablet_bootstrap.cc:492] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: Bootstrap starting.
I20260620 15:00:29.244236 6927 tablet_bootstrap.cc:654] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:29.245651 6927 log.cc:824] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:29.246728 6928 tablet_bootstrap.cc:654] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:29.247066 6929 tablet_bootstrap.cc:492] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: Bootstrap starting.
I20260620 15:00:29.248001 6928 log.cc:824] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:29.249150 6927 tablet_bootstrap.cc:492] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: No bootstrap required, opened a new log
I20260620 15:00:29.249409 6927 ts_tablet_manager.cc:1397] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260620 15:00:29.250448 6929 tablet_bootstrap.cc:654] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:29.251600 6928 tablet_bootstrap.cc:492] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: No bootstrap required, opened a new log
I20260620 15:00:29.251631 6929 log.cc:824] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:29.252238 6928 ts_tablet_manager.cc:1397] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 15:00:29.254603 6929 tablet_bootstrap.cc:492] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: No bootstrap required, opened a new log
I20260620 15:00:29.254853 6929 ts_tablet_manager.cc:1397] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260620 15:00:29.257006 6927 raft_consensus.cc:348] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.257354 6927 raft_consensus.cc:374] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:29.257468 6927 raft_consensus.cc:729] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c319b528070b4e56a9dc6846f3e2db82, State: Initialized, Role: FOLLOWER
I20260620 15:00:29.258106 6927 consensus_queue.cc:260] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.262333 6909 heartbeater.cc:503] Master 127.2.70.62:36389 was elected leader, sending a full tablet report...
I20260620 15:00:29.263091 6927 ts_tablet_manager.cc:1428] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 15:00:29.263175 6928 raft_consensus.cc:348] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.263639 6928 raft_consensus.cc:374] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:29.263752 6928 raft_consensus.cc:729] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a62656f90ef45fdbd14377269fb5456, State: Initialized, Role: FOLLOWER
I20260620 15:00:29.264344 6928 consensus_queue.cc:260] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.265429 6777 heartbeater.cc:503] Master 127.2.70.62:36389 was elected leader, sending a full tablet report...
I20260620 15:00:29.266067 6928 ts_tablet_manager.cc:1428] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260620 15:00:29.265097 6929 raft_consensus.cc:348] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.266386 6929 raft_consensus.cc:374] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:29.266494 6929 raft_consensus.cc:729] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 467d39b0935e40e4ae5e52be84ebc572, State: Initialized, Role: FOLLOWER
I20260620 15:00:29.267068 6929 consensus_queue.cc:260] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.270195 6929 ts_tablet_manager.cc:1428] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260620 15:00:29.288667 6935 raft_consensus.cc:482] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:29.288932 6935 raft_consensus.cc:504] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.290784 6935 leader_election.cc:288] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c319b528070b4e56a9dc6846f3e2db82 (127.2.70.3:42273), 1a62656f90ef45fdbd14377269fb5456 (127.2.70.2:37217)
I20260620 15:00:29.297750 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.291980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:58737 (local address 127.2.70.2:37217)
0620 15:00:29.292214 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.292220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.292524 (+ 304us) server_negotiation.cc:407] Connection header received
0620 15:00:29.292717 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.292722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.292807 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.292900 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:29.293369 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.294281 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.295978 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.296743 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.296906 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.297008 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.297322 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.297450 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:00:29.297863 6937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.291849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:37217 (local address 127.2.70.1:58737)
0620 15:00:29.292422 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.292436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:29.292577 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:29.292929 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:29.292932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:29.292952 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:29.293226 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.293233 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.294412 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.294417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.295829 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.295841 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.296995 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.297002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.297158 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.297182 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:00:29.297286 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:00:29.297422 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":289,"threads_started":1}
I20260620 15:00:29.297878 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.291198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:40829 (local address 127.2.70.3:42273)
0620 15:00:29.291437 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.291445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.291796 (+ 351us) server_negotiation.cc:407] Connection header received
0620 15:00:29.292005 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.292033 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.292126 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.292220 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:29.292853 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.293875 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.295912 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.296831 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.297004 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.297120 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.297439 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.297602 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:00:29.297955 6936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.290996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:42273 (local address 127.2.70.1:40829)
0620 15:00:29.291652 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:29.291667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:29.291846 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:29.292311 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:29.292316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:29.292356 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:29.292715 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.292724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.294012 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.294017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.295725 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:29.295738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.297094 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.297103 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:29.297266 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.297290 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:00:29.297405 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 15:00:29.297562 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":329,"threads_started":1}
I20260620 15:00:29.298934 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d315f8022414f50b13e613e76842a28" candidate_uuid: "467d39b0935e40e4ae5e52be84ebc572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a62656f90ef45fdbd14377269fb5456" is_pre_election: true
I20260620 15:00:29.299019 6864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d315f8022414f50b13e613e76842a28" candidate_uuid: "467d39b0935e40e4ae5e52be84ebc572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c319b528070b4e56a9dc6846f3e2db82" is_pre_election: true
I20260620 15:00:29.299448 6732 raft_consensus.cc:2454] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467d39b0935e40e4ae5e52be84ebc572 in term 0.
I20260620 15:00:29.299561 6864 raft_consensus.cc:2454] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 467d39b0935e40e4ae5e52be84ebc572 in term 0.
I20260620 15:00:29.300143 6534 leader_election.cc:302] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a62656f90ef45fdbd14377269fb5456, 467d39b0935e40e4ae5e52be84ebc572; no voters:
I20260620 15:00:29.300595 6935 raft_consensus.cc:2781] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:29.300769 6935 raft_consensus.cc:482] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:29.300886 6935 raft_consensus.cc:3037] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:29.303442 6935 raft_consensus.cc:504] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.304100 6935 leader_election.cc:288] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [CANDIDATE]: Term 1 election: Requested vote from peers c319b528070b4e56a9dc6846f3e2db82 (127.2.70.3:42273), 1a62656f90ef45fdbd14377269fb5456 (127.2.70.2:37217)
I20260620 15:00:29.304579 6864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d315f8022414f50b13e613e76842a28" candidate_uuid: "467d39b0935e40e4ae5e52be84ebc572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c319b528070b4e56a9dc6846f3e2db82"
I20260620 15:00:29.304692 6732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d315f8022414f50b13e613e76842a28" candidate_uuid: "467d39b0935e40e4ae5e52be84ebc572" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a62656f90ef45fdbd14377269fb5456"
I20260620 15:00:29.304806 6864 raft_consensus.cc:3037] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:29.304971 6732 raft_consensus.cc:3037] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:29.307271 6864 raft_consensus.cc:2454] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467d39b0935e40e4ae5e52be84ebc572 in term 1.
I20260620 15:00:29.307521 6732 raft_consensus.cc:2454] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 467d39b0935e40e4ae5e52be84ebc572 in term 1.
I20260620 15:00:29.307770 6534 leader_election.cc:302] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 467d39b0935e40e4ae5e52be84ebc572, c319b528070b4e56a9dc6846f3e2db82; no voters:
I20260620 15:00:29.308079 6935 raft_consensus.cc:2781] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:29.308617 6935 raft_consensus.cc:686] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 1 LEADER]: Becoming Leader. State: Replica: 467d39b0935e40e4ae5e52be84ebc572, State: Running, Role: LEADER
I20260620 15:00:29.309224 6935 consensus_queue.cc:237] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } }
I20260620 15:00:29.314425 6455 catalog_manager.cc:5495] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 reported cstate change: term changed from 0 to 1, leader changed from <none> to 467d39b0935e40e4ae5e52be84ebc572 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "467d39b0935e40e4ae5e52be84ebc572" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "467d39b0935e40e4ae5e52be84ebc572" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 45989 } health_report { overall_health: HEALTHY } } }
I20260620 15:00:29.347604 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.342607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.2.70.1:45989)
0620 15:00:29.342805 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.342809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.342823 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:00:29.342905 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.342909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.342978 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.343132 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:29.343512 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.344206 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.345544 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.345929 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.346070 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.346538 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:29.347047 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:29.347216 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.347265 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.347340 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:00:29.356177 6732 raft_consensus.cc:1261] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 1 FOLLOWER]: Refusing update from remote peer 467d39b0935e40e4ae5e52be84ebc572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:29.356197 6864 raft_consensus.cc:1261] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 1 FOLLOWER]: Refusing update from remote peer 467d39b0935e40e4ae5e52be84ebc572: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:29.357156 6938 consensus_queue.cc:1035] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a62656f90ef45fdbd14377269fb5456" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:29.357622 6935 consensus_queue.cc:1035] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c319b528070b4e56a9dc6846f3e2db82" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:29.368444 6941 mvcc.cc:205] Tried to move back new op lower bound from 7298939409832206336 to 7298939409655439360. Current Snapshot: MvccSnapshot[applied={T|T < 7298939409832206336}]
I20260620 15:00:29.383054 6456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:29.383392 6456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:29.387742 6456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cec8e6a8d77a4c5fa82c3f7f1f589d7b: Sending DeleteTablet for 3 replicas of tablet 8d315f8022414f50b13e613e76842a28
I20260620 15:00:29.389026 6844 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d315f8022414f50b13e613e76842a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:29 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:46948
I20260620 15:00:29.388984 6712 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d315f8022414f50b13e613e76842a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:29 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:37764
I20260620 15:00:29.389303 6579 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d315f8022414f50b13e613e76842a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:29 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:58678
I20260620 15:00:29.391259 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 6517
I20260620 15:00:29.400655 6953 tablet_replica.cc:330] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: stopping tablet replica
I20260620 15:00:29.401458 6951 tablet_replica.cc:330] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: stopping tablet replica
I20260620 15:00:29.401916 6951 raft_consensus.cc:2229] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:29.402449 6951 raft_consensus.cc:2258] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:29.403035 6953 raft_consensus.cc:2229] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:29.403733 6953 raft_consensus.cc:2258] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:29.406134 6953 ts_tablet_manager.cc:1905] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:29.406787 6952 tablet_replica.cc:330] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: stopping tablet replica
I20260620 15:00:29.407193 6952 raft_consensus.cc:2229] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:29.407703 6952 raft_consensus.cc:2258] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:29.407972 6951 ts_tablet_manager.cc:1905] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:29.415313 6952 ts_tablet_manager.cc:1905] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:29.416801 6951 ts_tablet_manager.cc:1918] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:29.417042 6951 log.cc:1196] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/8d315f8022414f50b13e613e76842a28
I20260620 15:00:29.417776 6951 ts_tablet_manager.cc:1939] T 8d315f8022414f50b13e613e76842a28 P 1a62656f90ef45fdbd14377269fb5456: Deleting consensus metadata
I20260620 15:00:29.419950 6442 catalog_manager.cc:4841] TS 1a62656f90ef45fdbd14377269fb5456 (127.2.70.2:37217): tablet 8d315f8022414f50b13e613e76842a28 (table test-table [id=02521812c3f04a1c90ce28e6aaf6067e]) successfully deleted
Jun 20 15:00:29 dist-test-slave-n5vv krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:00:29.423187 6953 ts_tablet_manager.cc:1918] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:29.423496 6953 log.cc:1196] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/8d315f8022414f50b13e613e76842a28
I20260620 15:00:29.424203 6953 ts_tablet_manager.cc:1939] T 8d315f8022414f50b13e613e76842a28 P 467d39b0935e40e4ae5e52be84ebc572: Deleting consensus metadata
I20260620 15:00:29.425881 6952 ts_tablet_manager.cc:1918] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:29.426106 6952 log.cc:1196] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/8d315f8022414f50b13e613e76842a28
I20260620 15:00:29.426177 6442 catalog_manager.cc:4841] TS 467d39b0935e40e4ae5e52be84ebc572 (127.2.70.1:45989): tablet 8d315f8022414f50b13e613e76842a28 (table test-table [id=02521812c3f04a1c90ce28e6aaf6067e]) successfully deleted
I20260620 15:00:29.426733 6952 ts_tablet_manager.cc:1939] T 8d315f8022414f50b13e613e76842a28 P c319b528070b4e56a9dc6846f3e2db82: Deleting consensus metadata
I20260620 15:00:29.428428 6442 catalog_manager.cc:4841] TS c319b528070b4e56a9dc6846f3e2db82 (127.2.70.3:42273): tablet 8d315f8022414f50b13e613e76842a28 (table test-table [id=02521812c3f04a1c90ce28e6aaf6067e]) successfully deleted
I20260620 15:00:29.429631 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.393434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58700 (local address 127.2.70.1:45989)
0620 15:00:29.393677 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.393687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.408855 (+ 15168us) server_negotiation.cc:407] Connection header received
0620 15:00:29.409089 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.409095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.409189 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.409306 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:29.411182 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.412580 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.414677 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.415451 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.415570 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.421132 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:29.421145 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:29.421159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:29.421199 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:29.425853 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.426954 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.426963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.426968 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.427177 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.427656 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.427662 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.427666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.428104 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:29.428443 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.429053 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.429243 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:00:29.430308 6639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:29.528995 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 6517
I20260620 15:00:29.546749 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 6650
Jun 20 15:00:29 dist-test-slave-n5vv krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:00:29.564502 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.547795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.2.70.2:37217)
0620 15:00:29.548015 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.548022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.548045 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:00:29.548110 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.548116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.548195 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.548318 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:29.549440 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.550705 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.551864 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.552546 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.552677 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.559318 (+ 6641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:29.559332 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:29.559348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:29.559389 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:29.562298 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.562876 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.562883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.562885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.563028 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.563324 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.563327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.563329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.563695 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:29.563903 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.564112 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.564262 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:00:29.565137 6772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:29.676986 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 6650
I20260620 15:00:29.694169 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 6782
Jun 20 15:00:29 dist-test-slave-n5vv krb5kdc[6412](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:00:29.708298 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:29.694950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.2.70.3:42273)
0620 15:00:29.695130 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:29.695135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:29.695153 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:00:29.695262 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:29.695266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:29.695329 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:29.695410 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:29.696640 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.697344 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.698554 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:29.698930 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:29.699023 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:29.702956 (+ 3933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:29.702971 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:29.702999 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:29.703032 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:29.705884 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.706448 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.706454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.706457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.706580 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:29.706951 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:29.706954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:29.706956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:29.707316 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:29.707521 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:29.707768 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 15:00:29.707920 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:00:29.708915 6904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:29.799178 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 6782
I20260620 15:00:29.815573 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 6425
I20260620 15:00:29.816684 6486 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:29.922201 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 6425
2026-06-20T15:00:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4131 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T15:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:29Z Disabled control of system clock
I20260620 15:00:29.988433 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35115
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:35115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 15:00:30.310585 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:30.310879 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:30.310963 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:30.318219 6962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:30.318337 6962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:30.318379 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:30.318437 6962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:30.318480 6962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:30.327654 6962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:35115
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:30.328892 6962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:30.330628 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:30.342598 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:30.342527 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:30.342574 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:30.343863 6962 server_base.cc:1034] running on GCE node
I20260620 15:00:30.344714 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:30.346175 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:30.347378 6962 hybrid_clock.cc:648] HybridClock initialized: now 1781967630347306 us; error 70 us; skew 500 ppm
I20260620 15:00:30.347698 6962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:00:30.350041 6962 webserver.cc:460] Webserver started at http://127.2.70.62:37735/ using document root <none> and password file <none>
I20260620 15:00:30.350719 6962 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:30.350803 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:30.351078 6962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:30.353979 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a"
format_stamp: "Formatted at 2026-06-20 15:00:30 on dist-test-slave-n5vv"
server_key: "4599d573aa1a278e407ece36165b2cb0"
server_key_iv: "5aa968cbca2f301004366fab641f14a2"
server_key_version: "encryptionkey@0"
I20260620 15:00:30.354885 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a"
format_stamp: "Formatted at 2026-06-20 15:00:30 on dist-test-slave-n5vv"
server_key: "4599d573aa1a278e407ece36165b2cb0"
server_key_iv: "5aa968cbca2f301004366fab641f14a2"
server_key_version: "encryptionkey@0"
I20260620 15:00:30.360033 6962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:30.363469 6976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:30.364816 6962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:30.365044 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a"
format_stamp: "Formatted at 2026-06-20 15:00:30 on dist-test-slave-n5vv"
server_key: "4599d573aa1a278e407ece36165b2cb0"
server_key_iv: "5aa968cbca2f301004366fab641f14a2"
server_key_version: "encryptionkey@0"
I20260620 15:00:30.365279 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:30.371181 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:30.371956 6962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:30.372253 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:30.390682 6962 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:35115
I20260620 15:00:30.390702 7027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:35115 every 8 connection(s)
I20260620 15:00:30.392117 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:30.395584 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 6962
I20260620 15:00:30.395793 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:30.396034 7028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:30.396495 2328 external_mini_cluster.cc:1428] Setting key 6fb3ff5980300da46a54e41c3c71069a
I20260620 15:00:30.405565 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:30.398471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.2.70.62:35115)
0620 15:00:30.399527 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:30.399541 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:30.399583 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:00:30.401793 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:30.401818 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:30.402323 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:30.402672 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:30.403836 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:30.403842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:30.403856 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:30.403870 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:30.403957 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:30.403974 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:30.404150 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:30.404293 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 15:00:30.404567 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":405,"threads_started":1}
I20260620 15:00:30.409597 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a: Bootstrap starting.
I20260620 15:00:30.412740 7028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:30.413849 7028 log.cc:824] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:30.416519 7028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a: No bootstrap required, opened a new log
I20260620 15:00:30.423698 7028 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35115 } }
I20260620 15:00:30.424067 7028 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:30.424173 7028 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fe52bd178d4f4e8a48ab4a15cd264a, State: Initialized, Role: FOLLOWER
I20260620 15:00:30.424803 7028 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35115 } }
I20260620 15:00:30.425036 7028 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:30.425182 7028 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:30.425351 7028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:30.427505 7028 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35115 } }
I20260620 15:00:30.428088 7028 leader_election.cc:302] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1fe52bd178d4f4e8a48ab4a15cd264a; no voters:
I20260620 15:00:30.428702 7028 leader_election.cc:288] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:30.428828 7033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:30.429638 7033 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [term 1 LEADER]: Becoming Leader. State: Replica: c1fe52bd178d4f4e8a48ab4a15cd264a, State: Running, Role: LEADER
I20260620 15:00:30.430194 7033 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35115 } }
I20260620 15:00:30.430842 7028 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:30.432891 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1fe52bd178d4f4e8a48ab4a15cd264a. Latest consensus state: current_term: 1 leader_uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35115 } } }
I20260620 15:00:30.433197 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:30.433403 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1fe52bd178d4f4e8a48ab4a15cd264a" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35115 } } }
I20260620 15:00:30.433799 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:30.435039 7040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:30.440469 7040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:30.450187 7040 catalog_manager.cc:1347] Generated new cluster ID: 3bdefe1a470947f494815e7b3430636e
I20260620 15:00:30.450313 7040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:30.474753 7040 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:30.475169 7040 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:30.481029 7040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a: Generated new TSK 0
I20260620 15:00:30.481884 7040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:00:30.490768 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35115
--builtin_ntp_servers=127.2.70.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 15:00:30.842737 7052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:30.843081 7052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:30.843250 7052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:30.850728 7052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:30.850883 7052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:30.851070 7052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:30.860363 7052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:30.861865 7052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:30.863883 7052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:30.878082 7057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:30.878153 7058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:30.878115 7060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:30.879098 7052 server_base.cc:1034] running on GCE node
I20260620 15:00:30.879788 7052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:30.880797 7052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:30.882042 7052 hybrid_clock.cc:648] HybridClock initialized: now 1781967630881970 us; error 69 us; skew 500 ppm
I20260620 15:00:30.882393 7052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:00:30.884824 7052 webserver.cc:460] Webserver started at http://127.2.70.1:37767/ using document root <none> and password file <none>
I20260620 15:00:30.885654 7052 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:30.885787 7052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:30.886123 7052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:30.888994 7052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "1028873280b44660af3532d2bfbcecc1"
format_stamp: "Formatted at 2026-06-20 15:00:30 on dist-test-slave-n5vv"
server_key: "d99da8cf4c9873a3d2e854b2dfec29aa"
server_key_iv: "4ff403357897fb190e4a2d7dcb24d26d"
server_key_version: "encryptionkey@0"
I20260620 15:00:30.889843 7052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "1028873280b44660af3532d2bfbcecc1"
format_stamp: "Formatted at 2026-06-20 15:00:30 on dist-test-slave-n5vv"
server_key: "d99da8cf4c9873a3d2e854b2dfec29aa"
server_key_iv: "4ff403357897fb190e4a2d7dcb24d26d"
server_key_version: "encryptionkey@0"
I20260620 15:00:30.895118 7052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:30.898680 7066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:30.900027 7052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:00:30.900204 7052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "1028873280b44660af3532d2bfbcecc1"
format_stamp: "Formatted at 2026-06-20 15:00:30 on dist-test-slave-n5vv"
server_key: "d99da8cf4c9873a3d2e854b2dfec29aa"
server_key_iv: "4ff403357897fb190e4a2d7dcb24d26d"
server_key_version: "encryptionkey@0"
I20260620 15:00:30.900439 7052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:30.906419 7052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:30.907217 7052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:30.907495 7052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:30.908404 7052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:30.909821 7052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:30.909943 7052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:30.910082 7052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:30.910173 7052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:30.945982 7052 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:36107
I20260620 15:00:30.946034 7178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:36107 every 8 connection(s)
I20260620 15:00:30.947383 7052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:30.955392 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 7052
I20260620 15:00:30.955626 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:30.955914 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:30.950007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:39887 (local address 127.2.70.62:35115)
0620 15:00:30.950462 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:30.950467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:30.951725 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 15:00:30.953457 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:30.953462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:30.953569 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:30.953693 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:30.955113 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:30.955117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:30.955121 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:30.955136 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:30.955179 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:30.955191 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:30.955304 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:30.955568 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 15:00:30.955637 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":249,"threads_started":1}
I20260620 15:00:30.956421 2328 external_mini_cluster.cc:1428] Setting key f3b782e566b25989f8c27e98f5c60380
I20260620 15:00:30.957284 7182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:30.950550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35115 (local address 127.2.70.1:39887)
0620 15:00:30.951543 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:30.951581 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:30.953168 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:30.953960 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:30.953968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:30.954064 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:30.954764 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:00:30.954767 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:30.954803 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:00:30.954925 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:30.955307 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:30.955326 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:30.955335 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 15:00:30.955645 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 15:00:30.955961 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":271,"threads_started":1}
I20260620 15:00:30.958901 7179 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35115
I20260620 15:00:30.959223 7179 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:30.959343 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35115
--builtin_ntp_servers=127.2.70.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 15:00:30.959905 7179 heartbeater.cc:511] Master 127.2.70.62:35115 requested a full tablet report, sending...
I20260620 15:00:30.962644 6992 ts_manager.cc:194] Registered new tserver with Master: 1028873280b44660af3532d2bfbcecc1 (127.2.70.1:36107)
W20260620 15:00:31.284952 7183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:31.285274 7183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:31.285574 7183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:31.293081 7183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:31.293231 7183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:31.293450 7183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:31.303215 7183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:31.304664 7183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:31.306618 7183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:31.320370 7189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:31.320472 7191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:31.320429 7188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:31.326877 7183 server_base.cc:1034] running on GCE node
I20260620 15:00:31.327498 7183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:31.328536 7183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:31.329759 7183 hybrid_clock.cc:648] HybridClock initialized: now 1781967631329704 us; error 54 us; skew 500 ppm
I20260620 15:00:31.330084 7183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:00:31.332460 7183 webserver.cc:460] Webserver started at http://127.2.70.2:38839/ using document root <none> and password file <none>
I20260620 15:00:31.333209 7183 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:31.333343 7183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:31.333742 7183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:31.336784 7183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "0e412db76e15446c99bb48bce8699ecd"
format_stamp: "Formatted at 2026-06-20 15:00:31 on dist-test-slave-n5vv"
server_key: "b3f6660eeab98cf455483055b33d2cfb"
server_key_iv: "e0bce590b6fc32c9458c0013aabdaa3a"
server_key_version: "encryptionkey@0"
I20260620 15:00:31.337625 7183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "0e412db76e15446c99bb48bce8699ecd"
format_stamp: "Formatted at 2026-06-20 15:00:31 on dist-test-slave-n5vv"
server_key: "b3f6660eeab98cf455483055b33d2cfb"
server_key_iv: "e0bce590b6fc32c9458c0013aabdaa3a"
server_key_version: "encryptionkey@0"
I20260620 15:00:31.342695 7183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:00:31.346143 7197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:31.347528 7183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:00:31.347694 7183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "0e412db76e15446c99bb48bce8699ecd"
format_stamp: "Formatted at 2026-06-20 15:00:31 on dist-test-slave-n5vv"
server_key: "b3f6660eeab98cf455483055b33d2cfb"
server_key_iv: "e0bce590b6fc32c9458c0013aabdaa3a"
server_key_version: "encryptionkey@0"
I20260620 15:00:31.347927 7183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:31.353693 7183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:31.354455 7183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:31.354727 7183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:31.355603 7183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:31.356927 7183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:31.357065 7183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:31.357191 7183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:31.357280 7183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:31.391804 7183 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:33767
I20260620 15:00:31.391822 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:33767 every 8 connection(s)
I20260620 15:00:31.393427 7183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:31.398638 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 7183
I20260620 15:00:31.398833 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:31.399384 2328 external_mini_cluster.cc:1428] Setting key 99dc4c24c093a6de7f621a7f991706d1
I20260620 15:00:31.402782 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35115
--builtin_ntp_servers=127.2.70.20:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 15:00:31.403004 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.396108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:58801 (local address 127.2.70.62:35115)
0620 15:00:31.396279 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:31.396283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:31.397946 (+ 1663us) server_negotiation.cc:407] Connection header received
0620 15:00:31.400350 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:31.400357 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:31.400458 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:31.400559 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:31.402231 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:31.402237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:31.402242 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:31.402261 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:31.402298 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:31.402312 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:31.402425 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:31.402667 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 15:00:31.402723 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:00:31.404340 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.396777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35115 (local address 127.2.70.2:58801)
0620 15:00:31.397760 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:31.397795 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:31.400049 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:31.400823 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:31.400835 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:31.400956 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:31.401877 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:00:31.401881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:31.401932 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:00:31.402045 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:31.402434 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:31.402446 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:31.402457 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:00:31.402740 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 15:00:31.403009 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":292,"threads_started":1}
I20260620 15:00:31.406567 7310 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35115
I20260620 15:00:31.406865 7310 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:31.407523 7310 heartbeater.cc:511] Master 127.2.70.62:35115 requested a full tablet report, sending...
I20260620 15:00:31.409106 6992 ts_manager.cc:194] Registered new tserver with Master: 0e412db76e15446c99bb48bce8699ecd (127.2.70.2:33767)
W20260620 15:00:31.731866 7313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:31.732239 7313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:31.732424 7313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:31.740931 7313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:31.741074 7313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:31.741268 7313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:31.752441 7313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:31.753891 7313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:31.755676 7313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:31.769014 7318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:31.769040 7319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:31.769078 7321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:31.769917 7313 server_base.cc:1034] running on GCE node
I20260620 15:00:31.770535 7313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:31.771552 7313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:31.772781 7313 hybrid_clock.cc:648] HybridClock initialized: now 1781967631772710 us; error 64 us; skew 500 ppm
I20260620 15:00:31.773108 7313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:00:31.775571 7313 webserver.cc:460] Webserver started at http://127.2.70.3:35667/ using document root <none> and password file <none>
I20260620 15:00:31.776261 7313 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:31.776381 7313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:31.776703 7313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:31.779515 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef"
format_stamp: "Formatted at 2026-06-20 15:00:31 on dist-test-slave-n5vv"
server_key: "26b16d8024c5b290ff73d243162d16dc"
server_key_iv: "16012b58e099acf839c03d04fdbc7249"
server_key_version: "encryptionkey@0"
I20260620 15:00:31.780269 7313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef"
format_stamp: "Formatted at 2026-06-20 15:00:31 on dist-test-slave-n5vv"
server_key: "26b16d8024c5b290ff73d243162d16dc"
server_key_iv: "16012b58e099acf839c03d04fdbc7249"
server_key_version: "encryptionkey@0"
I20260620 15:00:31.785255 7313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:00:31.788622 7327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:31.789943 7313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:31.790110 7313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef"
format_stamp: "Formatted at 2026-06-20 15:00:31 on dist-test-slave-n5vv"
server_key: "26b16d8024c5b290ff73d243162d16dc"
server_key_iv: "16012b58e099acf839c03d04fdbc7249"
server_key_version: "encryptionkey@0"
I20260620 15:00:31.790333 7313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:31.795792 7313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:31.796592 7313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:31.796845 7313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:31.797772 7313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:31.799060 7313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:31.799177 7313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:31.799306 7313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:31.799394 7313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:31.832768 7313 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:46373
I20260620 15:00:31.832810 7439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:46373 every 8 connection(s)
I20260620 15:00:31.834105 7313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:31.842222 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.836704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:51203 (local address 127.2.70.62:35115)
0620 15:00:31.836918 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:31.836925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:31.838379 (+ 1454us) server_negotiation.cc:407] Connection header received
0620 15:00:31.840028 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:31.840032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:31.840085 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:31.840175 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:31.841620 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:31.841624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:31.841627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:31.841644 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:31.841677 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:31.841689 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:31.841790 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:31.841961 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 15:00:31.842014 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:00:31.842970 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 7313
I20260620 15:00:31.843166 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:31.843374 7442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.837295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35115 (local address 127.2.70.3:51203)
0620 15:00:31.838217 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:31.838251 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:31.839777 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:31.840400 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:31.840429 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:31.840520 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:31.841233 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:00:31.841235 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:31.841271 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:00:31.841367 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:31.841780 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:31.841787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:31.841793 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:00:31.842007 (+ 214us) client_negotiation.cc:240] Negotiation successful
0620 15:00:31.842196 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":323,"threads_started":1}
I20260620 15:00:31.843686 2328 external_mini_cluster.cc:1428] Setting key 0c9b47aa0eef98bad559f8693c073cf6
I20260620 15:00:31.844924 7440 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35115
I20260620 15:00:31.845256 7440 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:31.845940 7440 heartbeater.cc:511] Master 127.2.70.62:35115 requested a full tablet report, sending...
I20260620 15:00:31.847337 6992 ts_manager.cc:194] Registered new tserver with Master: 91d0e1e4f9704b0aac1b1cd7a06733ef (127.2.70.3:46373)
I20260620 15:00:31.859087 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:00:31.866256 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.863573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.2.70.62:35115)
0620 15:00:31.863737 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:31.863742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:31.864271 (+ 529us) server_negotiation.cc:407] Connection header received
0620 15:00:31.864471 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:31.864476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:31.864546 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:31.864651 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:31.865664 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:31.865669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:31.865672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:31.865687 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:31.865717 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:31.865729 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:31.865856 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:31.865934 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 15:00:31.865998 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:00:31.870419 6992 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:31.877461 6992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:31.898905 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.890431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.2.70.1:36107)
0620 15:00:31.892724 (+ 2293us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:31.892746 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:31.894050 (+ 1304us) server_negotiation.cc:407] Connection header received
0620 15:00:31.894144 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:31.894157 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:31.894636 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:31.894816 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:31.897638 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:31.897650 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:31.897666 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:31.897693 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:31.897752 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:31.897774 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:31.897892 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:31.898440 (+ 548us) server_negotiation.cc:299] Negotiation successful
0620 15:00:31.898549 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2087,"thread_start_us":1921,"threads_started":1}
I20260620 15:00:31.898989 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.889819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:36107 (local address 127.0.0.1:41214)
0620 15:00:31.893720 (+ 3901us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:31.893792 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:31.893910 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:31.896113 (+ 2203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:31.896124 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:31.896241 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:31.897298 (+ 1057us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:00:31.897301 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:31.897336 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:00:31.897434 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:31.897942 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:31.897952 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:31.897959 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:00:31.898117 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 15:00:31.898219 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3744,"thread_start_us":3083,"threads_started":1}
I20260620 15:00:31.900969 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.889826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.2.70.2:33767)
0620 15:00:31.897737 (+ 7911us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:31.897746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:31.897772 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:00:31.897837 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:31.897843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:31.898084 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:31.899192 (+ 1108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:31.900150 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:31.900157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:31.900169 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:31.900185 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:31.900238 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:31.900266 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:31.900431 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:31.900496 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 15:00:31.900594 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7167,"thread_start_us":7058,"threads_started":1}
I20260620 15:00:31.900969 7452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.889722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:33767 (local address 127.0.0.1:41510)
0620 15:00:31.891751 (+ 2029us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:31.891810 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:31.891961 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:31.898257 (+ 6296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:31.898261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:31.898281 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:31.899042 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:00:31.899045 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:31.899060 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:00:31.899105 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:31.900360 (+ 1255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:31.900365 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:31.900369 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 15:00:31.900506 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 15:00:31.900555 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1629,"thread_start_us":1447,"threads_started":1}
I20260620 15:00:31.902753 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.893334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.2.70.3:46373)
0620 15:00:31.899205 (+ 5871us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:31.899223 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:31.899255 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:00:31.899325 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:31.899335 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:31.899527 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:31.899695 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:31.901952 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:31.901960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:31.901972 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:31.901998 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:31.902054 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:31.902073 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:31.902237 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:31.902302 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 15:00:31.902392 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5701,"thread_start_us":5210,"threads_started":1}
I20260620 15:00:31.902833 7456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:31.892639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:46373 (local address 127.0.0.1:40276)
0620 15:00:31.897440 (+ 4801us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:31.897456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:31.897569 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:31.899683 (+ 2114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:31.899689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:31.899711 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:31.901716 (+ 2005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:00:31.901719 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:31.901734 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:00:31.901774 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:31.902189 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:31.902195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:31.902201 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:00:31.902328 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:00:31.902381 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4688,"thread_start_us":4538,"threads_started":1}
I20260620 15:00:31.903390 7114 tablet_service.cc:1467] Processing CreateTablet for tablet f2348e86ba804fe9b6e15a4f8b757657 (DEFAULT_TABLE table=test-table [id=a77dfb8983f04ac7b732050ac0f01496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:31.905006 7114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2348e86ba804fe9b6e15a4f8b757657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:31.904757 7245 tablet_service.cc:1467] Processing CreateTablet for tablet f2348e86ba804fe9b6e15a4f8b757657 (DEFAULT_TABLE table=test-table [id=a77dfb8983f04ac7b732050ac0f01496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:31.906057 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2348e86ba804fe9b6e15a4f8b757657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:31.906275 7375 tablet_service.cc:1467] Processing CreateTablet for tablet f2348e86ba804fe9b6e15a4f8b757657 (DEFAULT_TABLE table=test-table [id=a77dfb8983f04ac7b732050ac0f01496]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:31.907553 7375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2348e86ba804fe9b6e15a4f8b757657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:31.917408 7458 tablet_bootstrap.cc:492] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Bootstrap starting.
I20260620 15:00:31.920176 7459 tablet_bootstrap.cc:492] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Bootstrap starting.
I20260620 15:00:31.920485 7458 tablet_bootstrap.cc:654] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:31.922091 7460 tablet_bootstrap.cc:492] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Bootstrap starting.
I20260620 15:00:31.921890 7458 log.cc:824] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:31.923692 7459 tablet_bootstrap.cc:654] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:31.924952 7459 log.cc:824] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:31.925158 7460 tablet_bootstrap.cc:654] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:31.925678 7458 tablet_bootstrap.cc:492] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: No bootstrap required, opened a new log
I20260620 15:00:31.925943 7458 ts_tablet_manager.cc:1397] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:00:31.926384 7460 log.cc:824] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:31.927872 7459 tablet_bootstrap.cc:492] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: No bootstrap required, opened a new log
I20260620 15:00:31.928118 7459 ts_tablet_manager.cc:1397] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260620 15:00:31.928715 7460 tablet_bootstrap.cc:492] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: No bootstrap required, opened a new log
I20260620 15:00:31.928961 7460 ts_tablet_manager.cc:1397] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 15:00:31.936257 7460 raft_consensus.cc:348] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:31.936317 7458 raft_consensus.cc:348] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:31.936623 7460 raft_consensus.cc:374] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:31.936712 7460 raft_consensus.cc:729] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91d0e1e4f9704b0aac1b1cd7a06733ef, State: Initialized, Role: FOLLOWER
I20260620 15:00:31.936717 7458 raft_consensus.cc:374] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:31.936812 7458 raft_consensus.cc:729] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1028873280b44660af3532d2bfbcecc1, State: Initialized, Role: FOLLOWER
I20260620 15:00:31.937309 7460 consensus_queue.cc:260] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:31.937381 7458 consensus_queue.cc:260] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:31.938452 7440 heartbeater.cc:503] Master 127.2.70.62:35115 was elected leader, sending a full tablet report...
I20260620 15:00:31.938923 7460 ts_tablet_manager.cc:1428] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260620 15:00:31.939399 7459 raft_consensus.cc:348] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:31.939742 7459 raft_consensus.cc:374] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:31.939853 7459 raft_consensus.cc:729] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e412db76e15446c99bb48bce8699ecd, State: Initialized, Role: FOLLOWER
I20260620 15:00:31.940765 7459 consensus_queue.cc:260] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:31.941897 7310 heartbeater.cc:503] Master 127.2.70.62:35115 was elected leader, sending a full tablet report...
I20260620 15:00:31.942504 7459 ts_tablet_manager.cc:1428] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260620 15:00:31.943382 7179 heartbeater.cc:503] Master 127.2.70.62:35115 was elected leader, sending a full tablet report...
I20260620 15:00:31.945314 7458 ts_tablet_manager.cc:1428] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Time spent starting tablet: real 0.019s user 0.013s sys 0.003s
W20260620 15:00:31.949640 7180 tablet.cc:2369] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:00:32.085750 7441 tablet.cc:2369] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:32.115556 7466 raft_consensus.cc:482] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:32.115867 7466 raft_consensus.cc:504] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:32.117640 7466 leader_election.cc:288] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1028873280b44660af3532d2bfbcecc1 (127.2.70.1:36107), 91d0e1e4f9704b0aac1b1cd7a06733ef (127.2.70.3:46373)
I20260620 15:00:32.120877 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.118055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:42591 (local address 127.2.70.1:36107)
0620 15:00:32.118231 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:32.118238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:32.118844 (+ 606us) server_negotiation.cc:407] Connection header received
0620 15:00:32.119079 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:32.119083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:32.119133 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:32.119235 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:32.120230 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:32.120236 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:32.120240 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:32.120262 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:32.120296 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:32.120307 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:32.120426 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:32.120528 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 15:00:32.120592 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:00:32.120965 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.118055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:48301 (local address 127.2.70.3:46373)
0620 15:00:32.118231 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:32.118238 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:32.119215 (+ 977us) server_negotiation.cc:407] Connection header received
0620 15:00:32.119392 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:32.119396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:32.119442 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:32.119547 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:32.120335 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:32.120341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:32.120345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:32.120364 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:32.120398 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:32.120408 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:32.120540 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:32.120639 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 15:00:32.120703 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:00:32.121124 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.117920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:36107 (local address 127.2.70.2:42591)
0620 15:00:32.118708 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:32.118730 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:32.118910 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:32.119261 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:32.119265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:32.119289 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:32.119994 (+ 705us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:00:32.119999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:32.120020 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:00:32.120070 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:32.120399 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:32.120404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:32.120416 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 15:00:32.120569 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 15:00:32.120618 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":471,"threads_started":1}
I20260620 15:00:32.121235 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.118004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:46373 (local address 127.2.70.2:48301)
0620 15:00:32.119101 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:32.119116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:32.119251 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:32.119567 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:32.119572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:32.119594 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:32.120156 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:00:32.120158 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:32.120170 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:00:32.120201 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:32.120498 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:32.120502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:32.120506 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 15:00:32.120676 (+ 170us) client_negotiation.cc:240] Negotiation successful
0620 15:00:32.120722 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"thread_start_us":781,"threads_started":1}
I20260620 15:00:32.122244 7395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2348e86ba804fe9b6e15a4f8b757657" candidate_uuid: "0e412db76e15446c99bb48bce8699ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" is_pre_election: true
I20260620 15:00:32.122347 7134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2348e86ba804fe9b6e15a4f8b757657" candidate_uuid: "0e412db76e15446c99bb48bce8699ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1028873280b44660af3532d2bfbcecc1" is_pre_election: true
I20260620 15:00:32.122748 7395 raft_consensus.cc:2454] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e412db76e15446c99bb48bce8699ecd in term 0.
I20260620 15:00:32.122843 7134 raft_consensus.cc:2454] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e412db76e15446c99bb48bce8699ecd in term 0.
I20260620 15:00:32.123422 7201 leader_election.cc:302] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e412db76e15446c99bb48bce8699ecd, 91d0e1e4f9704b0aac1b1cd7a06733ef; no voters:
I20260620 15:00:32.123793 7466 raft_consensus.cc:2781] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:32.123943 7466 raft_consensus.cc:482] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:32.124045 7466 raft_consensus.cc:3037] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:32.126240 7466 raft_consensus.cc:504] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:32.126821 7466 leader_election.cc:288] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [CANDIDATE]: Term 1 election: Requested vote from peers 1028873280b44660af3532d2bfbcecc1 (127.2.70.1:36107), 91d0e1e4f9704b0aac1b1cd7a06733ef (127.2.70.3:46373)
I20260620 15:00:32.127245 7134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2348e86ba804fe9b6e15a4f8b757657" candidate_uuid: "0e412db76e15446c99bb48bce8699ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1028873280b44660af3532d2bfbcecc1"
I20260620 15:00:32.127302 7395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2348e86ba804fe9b6e15a4f8b757657" candidate_uuid: "0e412db76e15446c99bb48bce8699ecd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef"
I20260620 15:00:32.127501 7395 raft_consensus.cc:3037] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:32.127482 7134 raft_consensus.cc:3037] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:32.129765 7395 raft_consensus.cc:2454] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e412db76e15446c99bb48bce8699ecd in term 1.
I20260620 15:00:32.129870 7134 raft_consensus.cc:2454] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e412db76e15446c99bb48bce8699ecd in term 1.
I20260620 15:00:32.130182 7201 leader_election.cc:302] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e412db76e15446c99bb48bce8699ecd, 91d0e1e4f9704b0aac1b1cd7a06733ef; no voters:
I20260620 15:00:32.130493 7466 raft_consensus.cc:2781] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:32.131021 7466 raft_consensus.cc:686] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 1 LEADER]: Becoming Leader. State: Replica: 0e412db76e15446c99bb48bce8699ecd, State: Running, Role: LEADER
I20260620 15:00:32.131511 7466 consensus_queue.cc:237] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } }
I20260620 15:00:32.136065 6993 catalog_manager.cc:5495] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e412db76e15446c99bb48bce8699ecd (127.2.70.2). New cstate: current_term: 1 leader_uuid: "0e412db76e15446c99bb48bce8699ecd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e412db76e15446c99bb48bce8699ecd" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 33767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 } health_report { overall_health: UNKNOWN } } }
W20260620 15:00:32.145448 7311 tablet.cc:2369] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:32.196537 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.194279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41516 (local address 127.2.70.2:33767)
0620 15:00:32.194484 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:32.194491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:32.194593 (+ 102us) server_negotiation.cc:407] Connection header received
0620 15:00:32.194838 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:32.194843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:32.194922 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:32.195034 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:32.195937 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:32.195943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:32.195948 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:32.195973 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:32.196005 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:32.196017 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:32.196149 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:32.196227 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 15:00:32.196283 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:00:32.204979 7395 raft_consensus.cc:1261] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 1 FOLLOWER]: Refusing update from remote peer 0e412db76e15446c99bb48bce8699ecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:32.204988 7134 raft_consensus.cc:1261] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 1 FOLLOWER]: Refusing update from remote peer 0e412db76e15446c99bb48bce8699ecd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:32.205953 7469 consensus_queue.cc:1035] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1028873280b44660af3532d2bfbcecc1" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 36107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:32.206334 7466 consensus_queue.cc:1035] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [LEADER]: Connected to new peer: Peer: permanent_uuid: "91d0e1e4f9704b0aac1b1cd7a06733ef" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:32.218149 7472 mvcc.cc:205] Tried to move back new op lower bound from 7298939421500559360 to 7298939421215080448. Current Snapshot: MvccSnapshot[applied={T|T < 7298939421500559360}]
I20260620 15:00:32.220896 7473 mvcc.cc:205] Tried to move back new op lower bound from 7298939421500559360 to 7298939421215080448. Current Snapshot: MvccSnapshot[applied={T|T < 7298939421500559360}]
I20260620 15:00:32.231691 6992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:32.231946 6992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:32.235657 6992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c1fe52bd178d4f4e8a48ab4a15cd264a: Sending DeleteTablet for 3 replicas of tablet f2348e86ba804fe9b6e15a4f8b757657
I20260620 15:00:32.236793 7245 tablet_service.cc:1514] Processing DeleteTablet for tablet f2348e86ba804fe9b6e15a4f8b757657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:32 UTC) from {username='slave'} at 127.0.0.1:41510
I20260620 15:00:32.236912 7114 tablet_service.cc:1514] Processing DeleteTablet for tablet f2348e86ba804fe9b6e15a4f8b757657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:32 UTC) from {username='slave'} at 127.0.0.1:41214
I20260620 15:00:32.237382 7375 tablet_service.cc:1514] Processing DeleteTablet for tablet f2348e86ba804fe9b6e15a4f8b757657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:32 UTC) from {username='slave'} at 127.0.0.1:40276
I20260620 15:00:32.239799 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 7052
I20260620 15:00:32.252473 7482 tablet_replica.cc:330] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: stopping tablet replica
I20260620 15:00:32.253131 7483 tablet_replica.cc:330] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: stopping tablet replica
I20260620 15:00:32.253942 7482 raft_consensus.cc:2229] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:32.254463 7483 raft_consensus.cc:2229] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:32.255324 7483 raft_consensus.cc:2258] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:32.255932 7482 raft_consensus.cc:2258] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:32.257907 7484 tablet_replica.cc:330] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: stopping tablet replica
I20260620 15:00:32.258311 7484 raft_consensus.cc:2229] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:32.258790 7484 raft_consensus.cc:2258] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:32.258870 7482 ts_tablet_manager.cc:1905] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:32.259737 7483 ts_tablet_manager.cc:1905] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:32.260254 7455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.241821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.2.70.1:36107)
0620 15:00:32.242048 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:32.242058 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:32.256238 (+ 14180us) server_negotiation.cc:407] Connection header received
0620 15:00:32.256734 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:32.256740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:32.256820 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:32.256895 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:32.259458 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:32.259466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:32.259474 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:32.259511 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:32.259548 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:32.259564 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:32.259660 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:32.259901 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 15:00:32.259964 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:00:32.261226 7174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:32.263412 7484 ts_tablet_manager.cc:1905] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:32.267513 7483 ts_tablet_manager.cc:1918] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:32.267735 7483 log.cc:1196] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/f2348e86ba804fe9b6e15a4f8b757657
I20260620 15:00:32.268342 7483 ts_tablet_manager.cc:1939] T f2348e86ba804fe9b6e15a4f8b757657 P 0e412db76e15446c99bb48bce8699ecd: Deleting consensus metadata
I20260620 15:00:32.270560 6979 catalog_manager.cc:4841] TS 0e412db76e15446c99bb48bce8699ecd (127.2.70.2:33767): tablet f2348e86ba804fe9b6e15a4f8b757657 (table test-table [id=a77dfb8983f04ac7b732050ac0f01496]) successfully deleted
I20260620 15:00:32.273679 7484 ts_tablet_manager.cc:1918] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:32.273890 7484 log.cc:1196] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/f2348e86ba804fe9b6e15a4f8b757657
I20260620 15:00:32.274544 7484 ts_tablet_manager.cc:1939] T f2348e86ba804fe9b6e15a4f8b757657 P 91d0e1e4f9704b0aac1b1cd7a06733ef: Deleting consensus metadata
I20260620 15:00:32.276728 6980 catalog_manager.cc:4841] TS 91d0e1e4f9704b0aac1b1cd7a06733ef (127.2.70.3:46373): tablet f2348e86ba804fe9b6e15a4f8b757657 (table test-table [id=a77dfb8983f04ac7b732050ac0f01496]) successfully deleted
I20260620 15:00:32.372135 7482 ts_tablet_manager.cc:1918] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:32.372282 7482 log.cc:1196] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/f2348e86ba804fe9b6e15a4f8b757657
I20260620 15:00:32.372730 7482 ts_tablet_manager.cc:1939] T f2348e86ba804fe9b6e15a4f8b757657 P 1028873280b44660af3532d2bfbcecc1: Deleting consensus metadata
I20260620 15:00:32.374377 6977 catalog_manager.cc:4841] TS 1028873280b44660af3532d2bfbcecc1 (127.2.70.1:36107): tablet f2348e86ba804fe9b6e15a4f8b757657 (table test-table [id=a77dfb8983f04ac7b732050ac0f01496]) successfully deleted
I20260620 15:00:32.374598 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 7052
I20260620 15:00:32.393451 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 7183
I20260620 15:00:32.397327 7454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.394806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.2.70.2:33767)
0620 15:00:32.395028 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:32.395035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:32.395053 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:00:32.395130 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:32.395134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:32.395183 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:32.395299 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:32.396265 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:32.396271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:32.396276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:32.396293 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:32.396326 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:32.396340 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:32.396458 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:32.396953 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 15:00:32.397029 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 15:00:32.398056 7305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:32.513386 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 7183
I20260620 15:00:32.530632 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 7313
I20260620 15:00:32.536624 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:32.531420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.2.70.3:46373)
0620 15:00:32.531652 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:32.531658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:32.531678 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:00:32.531791 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:32.531800 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:32.531935 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:32.532057 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:32.535960 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:32.535965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:32.535968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:32.535985 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:32.536019 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:32.536030 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:32.536186 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:32.536286 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 15:00:32.536345 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260620 15:00:32.537220 7435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:32.622861 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 7313
I20260620 15:00:32.639350 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 6962
I20260620 15:00:32.640580 7023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:32.744316 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 6962
2026-06-20T15:00:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2819 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 15:00:32.784927 2328 mini_oidc.cc:152] Starting JWKS server
I20260620 15:00:32.787535 2328 webserver.cc:460] Webserver started at https://127.0.0.1:39763/ using document root <none> and password file <none>
I20260620 15:00:32.787837 2328 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:00:32.788496 2328 webserver.cc:460] Webserver started at http://127.0.0.1:36375/ using document root <none> and password file <none>
2026-06-20T15:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:32Z Disabled control of system clock
I20260620 15:00:32.814059 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45987
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:45987
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39763/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/testchainca.cert with env {}
W20260620 15:00:33.135794 7495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:33.136128 7495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:33.136286 7495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:33.143538 7495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:33.143678 7495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:33.143770 7495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:33.143877 7495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:33.143965 7495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:00:33.144054 7495 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:00:33.144133 7495 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39763/jwks.json
I20260620 15:00:33.153848 7495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:45987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39763/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:33.155138 7495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:33.156450 7495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:33.168167 7500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:33.168313 7503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:33.168342 7501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:33.169267 7495 server_base.cc:1034] running on GCE node
I20260620 15:00:33.170099 7495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:33.171451 7495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:33.172675 7495 hybrid_clock.cc:648] HybridClock initialized: now 1781967633172619 us; error 47 us; skew 500 ppm
I20260620 15:00:33.173026 7495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:00:33.175184 7495 webserver.cc:460] Webserver started at http://127.2.70.62:40975/ using document root <none> and password file <none>
I20260620 15:00:33.175858 7495 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:33.175999 7495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:33.176308 7495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:33.179309 7495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "5ca2ee4c3b60431c956ec5f22fad2859"
format_stamp: "Formatted at 2026-06-20 15:00:33 on dist-test-slave-n5vv"
server_key: "2aca5671e69c66f03961b5eeadf1f2ac"
server_key_iv: "d78e4ae7e59e58142bb214665bc1dc30"
server_key_version: "encryptionkey@0"
I20260620 15:00:33.180146 7495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "5ca2ee4c3b60431c956ec5f22fad2859"
format_stamp: "Formatted at 2026-06-20 15:00:33 on dist-test-slave-n5vv"
server_key: "2aca5671e69c66f03961b5eeadf1f2ac"
server_key_iv: "d78e4ae7e59e58142bb214665bc1dc30"
server_key_version: "encryptionkey@0"
I20260620 15:00:33.185067 7495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:00:33.188504 7509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:33.189877 7495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:33.190066 7495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "5ca2ee4c3b60431c956ec5f22fad2859"
format_stamp: "Formatted at 2026-06-20 15:00:33 on dist-test-slave-n5vv"
server_key: "2aca5671e69c66f03961b5eeadf1f2ac"
server_key_iv: "d78e4ae7e59e58142bb214665bc1dc30"
server_key_version: "encryptionkey@0"
I20260620 15:00:33.190275 7495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:33.213203 7495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:33.214092 7495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:33.214399 7495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:33.232774 7495 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:45987
I20260620 15:00:33.232792 7560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:45987 every 8 connection(s)
I20260620 15:00:33.234189 7495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:33.238025 7561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:33.241098 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 7495
I20260620 15:00:33.241289 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:33.241956 2328 external_mini_cluster.cc:1428] Setting key 00e07c5bccb64cda134b9fc487dbd886
I20260620 15:00:33.250434 7561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859: Bootstrap starting.
I20260620 15:00:33.254132 7561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:33.255270 7561 log.cc:824] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:33.255666 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:33.243906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.2.70.62:45987)
0620 15:00:33.244827 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:33.244837 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:33.244878 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:00:33.246497 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:33.246519 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:33.247001 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:33.247325 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:33.247973 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:33.249300 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:33.251142 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:33.251886 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:33.252058 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:33.253310 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:33.253317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:33.253328 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:33.253928 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:33.254030 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:33.254050 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:33.254159 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:33.254396 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 15:00:33.254696 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":345,"threads_started":1}
I20260620 15:00:33.258442 7561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859: No bootstrap required, opened a new log
I20260620 15:00:33.259761 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:00:33.266949 7561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca2ee4c3b60431c956ec5f22fad2859" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45987 } }
I20260620 15:00:33.267268 7561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:33.267338 7561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca2ee4c3b60431c956ec5f22fad2859, State: Initialized, Role: FOLLOWER
I20260620 15:00:33.267804 7561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca2ee4c3b60431c956ec5f22fad2859" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45987 } }
I20260620 15:00:33.267985 7561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:33.268074 7561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:33.268230 7561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:33.270664 7561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca2ee4c3b60431c956ec5f22fad2859" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45987 } }
I20260620 15:00:33.271090 7561 leader_election.cc:302] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ca2ee4c3b60431c956ec5f22fad2859; no voters:
I20260620 15:00:33.271589 7561 leader_election.cc:288] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:33.271741 7566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:33.272454 7566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [term 1 LEADER]: Becoming Leader. State: Replica: 5ca2ee4c3b60431c956ec5f22fad2859, State: Running, Role: LEADER
I20260620 15:00:33.273042 7566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca2ee4c3b60431c956ec5f22fad2859" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45987 } }
I20260620 15:00:33.273722 7561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:33.276124 7568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ca2ee4c3b60431c956ec5f22fad2859. Latest consensus state: current_term: 1 leader_uuid: "5ca2ee4c3b60431c956ec5f22fad2859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca2ee4c3b60431c956ec5f22fad2859" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45987 } } }
I20260620 15:00:33.276576 7568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:33.276625 7567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ca2ee4c3b60431c956ec5f22fad2859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ca2ee4c3b60431c956ec5f22fad2859" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45987 } } }
I20260620 15:00:33.277801 7575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:33.277823 7567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:33.285442 7575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:33.293573 7575 catalog_manager.cc:1347] Generated new cluster ID: 9bb79023b33b44dbb9fed5cdd3a966e1
I20260620 15:00:33.294101 7575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:33.302578 7575 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:33.303783 7575 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:33.309192 7575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ca2ee4c3b60431c956ec5f22fad2859: Generated new TSK 0
I20260620 15:00:33.309990 7575 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 15:00:33.351250 7591 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 15:00:33.351428 7591 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:00:33.352326 7564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:33.324065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.2.70.62:45987)
0620 15:00:33.324236 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:33.324241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:33.324810 (+ 569us) server_negotiation.cc:407] Connection header received
0620 15:00:33.325185 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:33.325189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:33.325260 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:33.325431 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:00:33.326043 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:33.326839 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:33.328435 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:33.328922 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:33.329074 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:33.329660 (+ 586us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:00:33.351624 (+ 21964us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 15:00:33.352007 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":60}
W20260620 15:00:33.352468 7564 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41374: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 15:00:33.353956 7590 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:33.323961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45987 (local address 127.0.0.1:41374)
0620 15:00:33.324691 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:33.324712 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:33.324928 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:33.325463 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:33.325468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:33.325510 (+ 42us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:00:33.325877 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:33.325887 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:33.326996 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:33.327003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:33.328275 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:33.328286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:33.329052 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:33.329120 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:33.329274 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:33.329332 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:00:33.351944 (+ 22612us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 15:00:33.352298 (+ 354us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.70.62:45987: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":420,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.70.62:45987: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39763/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 15:00:33.355912 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 7495
I20260620 15:00:33.357211 7556 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:33.448247 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 7495
2026-06-20T15:00:33Z chronyd exiting
I20260620 15:00:33.498162 2328 test_util.cc:175] -----------------------------------------------
I20260620 15:00:33.498283 2328 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781967589245963-2328-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (717 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 15:00:33.501858 2328 mini_oidc.cc:152] Starting JWKS server
I20260620 15:00:33.503564 2328 webserver.cc:460] Webserver started at https://127.0.0.1:38205/ using document root <none> and password file <none>
I20260620 15:00:33.503758 2328 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:00:33.504302 2328 webserver.cc:460] Webserver started at http://127.0.0.1:38699/ using document root <none> and password file <none>
2026-06-20T15:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:33Z Disabled control of system clock
I20260620 15:00:33.532356 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:40437
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:40437
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38205/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/testchainca.cert with env {}
W20260620 15:00:33.854432 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:33.854723 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:33.854902 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:33.862182 7598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:33.862298 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:33.862401 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:33.862504 7598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:33.862567 7598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:00:33.862655 7598 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:00:33.862715 7598 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38205/jwks.json
I20260620 15:00:33.873160 7598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:40437
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:40437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38205/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:33.874536 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:33.875831 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:33.887944 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:33.888082 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:33.888056 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:33.889011 7598 server_base.cc:1034] running on GCE node
I20260620 15:00:33.889899 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:33.891266 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:33.892477 7598 hybrid_clock.cc:648] HybridClock initialized: now 1781967633892415 us; error 56 us; skew 500 ppm
I20260620 15:00:33.892828 7598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:00:33.895103 7598 webserver.cc:460] Webserver started at http://127.2.70.62:37869/ using document root <none> and password file <none>
I20260620 15:00:33.895854 7598 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:33.895969 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:33.896282 7598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:33.899173 7598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "d88833cd07ba40d3ad623450606c1ec7"
format_stamp: "Formatted at 2026-06-20 15:00:33 on dist-test-slave-n5vv"
server_key: "880dd7e40170886c88d9c486f724a2e7"
server_key_iv: "ca4065455cef24fa7e8386eaa79df87f"
server_key_version: "encryptionkey@0"
I20260620 15:00:33.900050 7598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "d88833cd07ba40d3ad623450606c1ec7"
format_stamp: "Formatted at 2026-06-20 15:00:33 on dist-test-slave-n5vv"
server_key: "880dd7e40170886c88d9c486f724a2e7"
server_key_iv: "ca4065455cef24fa7e8386eaa79df87f"
server_key_version: "encryptionkey@0"
I20260620 15:00:33.905212 7598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:00:33.908812 7612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:33.910334 7598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:33.910533 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "d88833cd07ba40d3ad623450606c1ec7"
format_stamp: "Formatted at 2026-06-20 15:00:33 on dist-test-slave-n5vv"
server_key: "880dd7e40170886c88d9c486f724a2e7"
server_key_iv: "ca4065455cef24fa7e8386eaa79df87f"
server_key_version: "encryptionkey@0"
I20260620 15:00:33.910755 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:33.941576 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:33.942423 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:33.942705 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:33.961040 7598 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:40437
I20260620 15:00:33.961064 7663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:40437 every 8 connection(s)
I20260620 15:00:33.962422 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:33.966125 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:33.970952 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 7598
I20260620 15:00:33.971264 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:33.971911 2328 external_mini_cluster.cc:1428] Setting key a227fdce2b5aa246a2f3eeacdd0e88cd
I20260620 15:00:33.976927 7664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7: Bootstrap starting.
I20260620 15:00:33.980111 7664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:33.981074 7664 log.cc:824] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:33.984364 7664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7: No bootstrap required, opened a new log
I20260620 15:00:33.985880 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:33.973952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53464 (local address 127.2.70.62:40437)
0620 15:00:33.975013 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:33.975028 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:33.975075 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:00:33.976680 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:33.976703 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:33.977221 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:33.977640 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:33.978248 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:33.979616 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:33.980925 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:33.981816 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:33.982048 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:33.983061 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:33.983070 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:33.983087 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:33.983817 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:33.983914 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:33.983940 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:33.984055 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:33.984236 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 15:00:33.984578 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":356,"threads_started":1}
I20260620 15:00:33.989699 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:00:33.992925 7664 raft_consensus.cc:348] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88833cd07ba40d3ad623450606c1ec7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40437 } }
I20260620 15:00:33.993286 7664 raft_consensus.cc:374] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:33.993394 7664 raft_consensus.cc:729] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d88833cd07ba40d3ad623450606c1ec7, State: Initialized, Role: FOLLOWER
I20260620 15:00:33.994005 7664 consensus_queue.cc:260] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88833cd07ba40d3ad623450606c1ec7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40437 } }
I20260620 15:00:33.994242 7664 raft_consensus.cc:388] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:33.994382 7664 raft_consensus.cc:482] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:33.994560 7664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:33.996917 7664 raft_consensus.cc:504] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88833cd07ba40d3ad623450606c1ec7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40437 } }
I20260620 15:00:33.997397 7664 leader_election.cc:302] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d88833cd07ba40d3ad623450606c1ec7; no voters:
I20260620 15:00:33.998076 7664 leader_election.cc:288] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:33.998200 7669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:33.998929 7669 raft_consensus.cc:686] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [term 1 LEADER]: Becoming Leader. State: Replica: d88833cd07ba40d3ad623450606c1ec7, State: Running, Role: LEADER
I20260620 15:00:33.999547 7669 consensus_queue.cc:237] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88833cd07ba40d3ad623450606c1ec7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40437 } }
I20260620 15:00:34.000372 7664 sys_catalog.cc:564] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:34.002506 7670 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d88833cd07ba40d3ad623450606c1ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88833cd07ba40d3ad623450606c1ec7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40437 } } }
I20260620 15:00:34.002912 7670 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:34.007438 7671 sys_catalog.cc:455] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d88833cd07ba40d3ad623450606c1ec7. Latest consensus state: current_term: 1 leader_uuid: "d88833cd07ba40d3ad623450606c1ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d88833cd07ba40d3ad623450606c1ec7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40437 } } }
I20260620 15:00:34.007786 7671 sys_catalog.cc:458] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:34.010650 7678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:34.015506 7678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:34.024133 7678 catalog_manager.cc:1347] Generated new cluster ID: c1ce44dd574d4d4cb7c9477ac73c01f3
I20260620 15:00:34.024267 7678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:34.036759 7678 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:34.038162 7678 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:34.048494 7678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d88833cd07ba40d3ad623450606c1ec7: Generated new TSK 0
I20260620 15:00:34.049361 7678 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 15:00:34.085074 7694 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 15:00:34.085206 7694 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:00:34.086531 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:34.065929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.2.70.62:40437)
0620 15:00:34.066131 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:34.066138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:34.066580 (+ 442us) server_negotiation.cc:407] Connection header received
0620 15:00:34.066784 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:34.066790 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:34.066878 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:34.067050 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:00:34.067514 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.068678 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.070119 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.070820 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.070946 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:34.071411 (+ 465us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:00:34.085815 (+ 14404us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38205/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 15:00:34.086229 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53478: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38205/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 15:00:34.086658 7667 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53478: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38205/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 15:00:34.086666 7693 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:34.065770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40437 (local address 127.0.0.1:53478)
0620 15:00:34.066479 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:34.066500 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:34.066621 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:34.067074 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:34.067077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:34.067086 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:00:34.067318 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:34.067326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.068820 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.068824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:34.069984 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:34.069997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.070941 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.070982 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:34.071116 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:34.071147 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:00:34.086082 (+ 14935us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38205/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 15:00:34.086318 (+ 236us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.70.62:40437: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38205/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":390,"threads_started":1}
I20260620 15:00:34.088547 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 7598
I20260620 15:00:34.089725 7659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:34.181404 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 7598
2026-06-20T15:00:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (734 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 15:00:34.236598 2328 mini_oidc.cc:152] Starting JWKS server
I20260620 15:00:34.238508 2328 webserver.cc:460] Webserver started at https://127.0.0.1:43203/ using document root <none> and password file <none>
I20260620 15:00:34.238749 2328 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 15:00:34.239362 2328 webserver.cc:460] Webserver started at http://127.0.0.1:42793/ using document root <none> and password file <none>
2026-06-20T15:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:34Z Disabled control of system clock
I20260620 15:00:34.267149 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42989
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:40533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:42989
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43203/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 15:00:34.593971 7701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:34.594261 7701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:34.594359 7701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:34.601533 7701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:34.601639 7701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:34.601675 7701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:34.601732 7701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:34.601774 7701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:00:34.601835 7701 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:00:34.601874 7701 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43203/jwks.json
I20260620 15:00:34.610805 7701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:42989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43203/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:34.612010 7701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:34.613760 7701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:34.625504 7706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:34.625442 7707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:34.625698 7709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:34.631011 7701 server_base.cc:1034] running on GCE node
I20260620 15:00:34.631788 7701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:34.633173 7701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:34.634371 7701 hybrid_clock.cc:648] HybridClock initialized: now 1781967634634290 us; error 71 us; skew 500 ppm
I20260620 15:00:34.634732 7701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:00:34.636927 7701 webserver.cc:460] Webserver started at http://127.2.70.62:41311/ using document root <none> and password file <none>
I20260620 15:00:34.637648 7701 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:34.637758 7701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:34.638073 7701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:34.640983 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "642467ea9047476696e93084cb49f936"
format_stamp: "Formatted at 2026-06-20 15:00:34 on dist-test-slave-n5vv"
server_key: "9b0a467240dc10a5048a2238435fda91"
server_key_iv: "fda2ca35184afd55da23eb2a82a60c0f"
server_key_version: "encryptionkey@0"
I20260620 15:00:34.641817 7701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "642467ea9047476696e93084cb49f936"
format_stamp: "Formatted at 2026-06-20 15:00:34 on dist-test-slave-n5vv"
server_key: "9b0a467240dc10a5048a2238435fda91"
server_key_iv: "fda2ca35184afd55da23eb2a82a60c0f"
server_key_version: "encryptionkey@0"
I20260620 15:00:34.646776 7701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 15:00:34.649976 7715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:34.651196 7701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:00:34.651365 7701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "642467ea9047476696e93084cb49f936"
format_stamp: "Formatted at 2026-06-20 15:00:34 on dist-test-slave-n5vv"
server_key: "9b0a467240dc10a5048a2238435fda91"
server_key_iv: "fda2ca35184afd55da23eb2a82a60c0f"
server_key_version: "encryptionkey@0"
I20260620 15:00:34.651575 7701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:34.667317 7701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:34.668102 7701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:34.668373 7701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:34.686726 7701 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:42989
I20260620 15:00:34.686774 7766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:42989 every 8 connection(s)
I20260620 15:00:34.688238 7701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:34.691911 7767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:34.697770 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 7701
I20260620 15:00:34.697957 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:34.698596 2328 external_mini_cluster.cc:1428] Setting key b1206c586af63a8f2ea008126975f0bb
I20260620 15:00:34.702778 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936: Bootstrap starting.
I20260620 15:00:34.705832 7767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:34.706799 7767 log.cc:824] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:34.709662 7767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936: No bootstrap required, opened a new log
I20260620 15:00:34.711411 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:34.700432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.2.70.62:42989)
0620 15:00:34.701441 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:34.701452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:34.701543 (+ 91us) server_negotiation.cc:407] Connection header received
0620 15:00:34.703194 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:34.703215 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:34.703662 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:34.704033 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:34.704576 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.705850 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.707114 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.707817 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.708024 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:34.709178 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:34.709185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:34.709198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:34.709780 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:34.709868 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:34.709888 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:34.710013 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:34.710245 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 15:00:34.710527 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":359,"threads_started":1}
I20260620 15:00:34.715158 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:00:34.718106 7767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642467ea9047476696e93084cb49f936" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42989 } }
I20260620 15:00:34.718478 7767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:34.718580 7767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 642467ea9047476696e93084cb49f936, State: Initialized, Role: FOLLOWER
I20260620 15:00:34.719162 7767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642467ea9047476696e93084cb49f936" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42989 } }
I20260620 15:00:34.719390 7767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:34.719523 7767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:34.719691 7767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:34.721845 7767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642467ea9047476696e93084cb49f936" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42989 } }
I20260620 15:00:34.722308 7767 leader_election.cc:302] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 642467ea9047476696e93084cb49f936; no voters:
I20260620 15:00:34.722839 7767 leader_election.cc:288] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:34.722977 7772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:34.724351 7772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [term 1 LEADER]: Becoming Leader. State: Replica: 642467ea9047476696e93084cb49f936, State: Running, Role: LEADER
I20260620 15:00:34.724979 7767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:34.724925 7772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642467ea9047476696e93084cb49f936" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42989 } }
I20260620 15:00:34.727473 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 642467ea9047476696e93084cb49f936. Latest consensus state: current_term: 1 leader_uuid: "642467ea9047476696e93084cb49f936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642467ea9047476696e93084cb49f936" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42989 } } }
I20260620 15:00:34.727839 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:34.728412 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "642467ea9047476696e93084cb49f936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "642467ea9047476696e93084cb49f936" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42989 } } }
I20260620 15:00:34.728778 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:34.728828 7781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:34.735440 7781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:34.744045 7781 catalog_manager.cc:1347] Generated new cluster ID: 52804d5cc582462fa7174e43cc280a6a
I20260620 15:00:34.744184 7781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:34.766794 7781 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:34.768106 7781 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:34.774923 7781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 642467ea9047476696e93084cb49f936: Generated new TSK 0
I20260620 15:00:34.776000 7781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:00:34.803766 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:34.795483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44654 (local address 127.2.70.62:42989)
0620 15:00:34.795684 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:34.795690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:34.796021 (+ 331us) server_negotiation.cc:407] Connection header received
0620 15:00:34.796228 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:34.796234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:34.796326 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:34.796525 (+ 199us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:00:34.796878 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.798056 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.799403 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.800090 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.800220 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:34.800691 (+ 471us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:00:34.803102 (+ 2411us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 15:00:34.803425 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 15:00:34.803892 7770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 15:00:34.805367 7797 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:34.795300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42989 (local address 127.0.0.1:44654)
0620 15:00:34.795922 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:34.795938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:34.796058 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:34.796497 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:34.796500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:34.796509 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:00:34.796743 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:34.796750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.798194 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.798198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:34.799276 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:34.799287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:34.800206 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:34.800273 (+ 67us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:34.800390 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:34.800425 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:00:34.803299 (+ 2874us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 15:00:34.804965 (+ 1666us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.70.62:42989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43203/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":1372,"thread_start_us":341,"threads_started":1}
I20260620 15:00:34.806669 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 7701
I20260620 15:00:34.807854 7762 generic_service.cc:196] Checking for leaks (request via RPC)
E20260620 15:00:34.816474 7798 webserver.cc:533] Webserver: sslize failed:
E20260620 15:00:34.816571 7798 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 15:00:34.899907 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 7701
2026-06-20T15:00:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T15:00:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:34Z Disabled control of system clock
I20260620 15:00:34.980681 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:40543
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:40543
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/testchainca.cert with env {}
W20260620 15:00:35.296288 7803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:35.296663 7803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:35.296773 7803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:35.303829 7803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:35.303937 7803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:35.303974 7803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:35.304032 7803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:35.304075 7803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 15:00:35.304128 7803 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 15:00:35.304167 7803 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 15:00:35.313851 7803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:40543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:40543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:35.315095 7803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:35.316751 7803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:35.328253 7809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:35.328313 7811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:35.328442 7808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:35.329556 7803 server_base.cc:1034] running on GCE node
I20260620 15:00:35.330355 7803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:35.331728 7803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:35.332929 7803 hybrid_clock.cc:648] HybridClock initialized: now 1781967635332868 us; error 64 us; skew 500 ppm
I20260620 15:00:35.333267 7803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:00:35.335651 7803 webserver.cc:460] Webserver started at http://127.2.70.62:42721/ using document root <none> and password file <none>
I20260620 15:00:35.336334 7803 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:35.336457 7803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:35.336764 7803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:35.339587 7803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "476cfd0b7cb84ab2ba3b25535e008990"
format_stamp: "Formatted at 2026-06-20 15:00:35 on dist-test-slave-n5vv"
server_key: "19cac4ea12e13eb10cb05f77975e6344"
server_key_iv: "492cb50ff1458db4f28538eb4879695e"
server_key_version: "encryptionkey@0"
I20260620 15:00:35.340360 7803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "476cfd0b7cb84ab2ba3b25535e008990"
format_stamp: "Formatted at 2026-06-20 15:00:35 on dist-test-slave-n5vv"
server_key: "19cac4ea12e13eb10cb05f77975e6344"
server_key_iv: "492cb50ff1458db4f28538eb4879695e"
server_key_version: "encryptionkey@0"
I20260620 15:00:35.345229 7803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:00:35.348624 7817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:35.350090 7803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:35.350263 7803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "476cfd0b7cb84ab2ba3b25535e008990"
format_stamp: "Formatted at 2026-06-20 15:00:35 on dist-test-slave-n5vv"
server_key: "19cac4ea12e13eb10cb05f77975e6344"
server_key_iv: "492cb50ff1458db4f28538eb4879695e"
server_key_version: "encryptionkey@0"
I20260620 15:00:35.350469 7803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:35.364895 7803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:35.365676 7803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:35.365939 7803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:35.383939 7803 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:40543
I20260620 15:00:35.383968 7868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:40543 every 8 connection(s)
I20260620 15:00:35.385247 7803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:35.388548 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 7803
I20260620 15:00:35.388782 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:35.389398 7869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:35.389611 2328 external_mini_cluster.cc:1428] Setting key 33e0eec038cb149b269a755dbd74496e
I20260620 15:00:35.401955 7869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990: Bootstrap starting.
I20260620 15:00:35.402547 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:35.391500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.2.70.62:40543)
0620 15:00:35.392441 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:35.392452 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:35.392492 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:00:35.394150 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:35.394171 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:35.394625 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:35.395005 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:35.395547 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:35.396828 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:35.398128 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:35.398798 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:35.398988 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:35.400217 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:35.400224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:35.400238 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:00:35.400764 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:35.400846 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:00:35.400866 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:35.401002 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:35.401205 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 15:00:35.401556 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":323,"threads_started":1}
I20260620 15:00:35.405192 7869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:35.406204 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:00:35.406387 7869 log.cc:824] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:35.409190 7869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990: No bootstrap required, opened a new log
I20260620 15:00:35.416595 7869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476cfd0b7cb84ab2ba3b25535e008990" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40543 } }
I20260620 15:00:35.417006 7869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:35.417080 7869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476cfd0b7cb84ab2ba3b25535e008990, State: Initialized, Role: FOLLOWER
I20260620 15:00:35.417654 7869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476cfd0b7cb84ab2ba3b25535e008990" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40543 } }
I20260620 15:00:35.417901 7869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:35.418047 7869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:35.418263 7869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:35.420539 7869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476cfd0b7cb84ab2ba3b25535e008990" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40543 } }
I20260620 15:00:35.421048 7869 leader_election.cc:302] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 476cfd0b7cb84ab2ba3b25535e008990; no voters:
I20260620 15:00:35.421716 7869 leader_election.cc:288] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:35.421864 7874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:35.422547 7874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [term 1 LEADER]: Becoming Leader. State: Replica: 476cfd0b7cb84ab2ba3b25535e008990, State: Running, Role: LEADER
I20260620 15:00:35.423107 7874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476cfd0b7cb84ab2ba3b25535e008990" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40543 } }
I20260620 15:00:35.423807 7869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:35.425952 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 476cfd0b7cb84ab2ba3b25535e008990. Latest consensus state: current_term: 1 leader_uuid: "476cfd0b7cb84ab2ba3b25535e008990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476cfd0b7cb84ab2ba3b25535e008990" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40543 } } }
I20260620 15:00:35.426373 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:35.426455 7875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "476cfd0b7cb84ab2ba3b25535e008990" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476cfd0b7cb84ab2ba3b25535e008990" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40543 } } }
I20260620 15:00:35.426750 7875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:35.430631 7882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:35.435313 7882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:35.443897 7882 catalog_manager.cc:1347] Generated new cluster ID: 63e82a9b9b3f4cd1bd4bf8678b50086c
I20260620 15:00:35.444022 7882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:35.465219 7882 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:35.466691 7882 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:35.479485 7882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 476cfd0b7cb84ab2ba3b25535e008990: Generated new TSK 0
I20260620 15:00:35.480526 7882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:00:35.528091 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:35.502488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.2.70.62:40543)
0620 15:00:35.502707 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:35.502713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:35.503129 (+ 416us) server_negotiation.cc:407] Connection header received
0620 15:00:35.503328 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:35.503332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:35.503425 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:35.503562 (+ 137us) server_negotiation.cc:226] Negotiated authn=JWT
0620 15:00:35.504129 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:35.504897 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:35.506996 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:35.507508 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:35.507611 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:35.508181 (+ 570us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 15:00:35.527547 (+ 19366us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 15:00:35.527864 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54494: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":80}
W20260620 15:00:35.528265 7872 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54494: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 15:00:35.528559 7898 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:35.502311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40543 (local address 127.0.0.1:54494)
0620 15:00:35.503000 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:35.503021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:35.503171 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:35.503598 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:35.503603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:35.503617 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0620 15:00:35.503963 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:35.503978 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:35.505048 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:35.505054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:35.506842 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:35.506858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:35.507636 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:35.507691 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:35.507874 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:35.507931 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 15:00:35.527831 (+ 19900us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 15:00:35.528153 (+ 322us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.70.62:40543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":308,"threads_started":1}
I20260620 15:00:35.529971 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 7803
I20260620 15:00:35.531147 7864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:35.626595 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 7803
2026-06-20T15:00:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (705 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (410 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (400 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:36 dist-test-slave-n5vv krb5kdc[7913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:36 dist-test-slave-n5vv krb5kdc[7913](info): set up 2 sockets
Jun 20 15:00:36 dist-test-slave-n5vv krb5kdc[7913](info): commencing operation
krb5kdc: starting...
W20260620 15:00:37.526147 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:37 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:37.734110 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35347
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:38.059806 7926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:38.060105 7926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:38.060256 7926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:38.067538 7926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:38.067641 7926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:38.067740 7926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:38.067816 7926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:38.067875 7926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:38.077961 7926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:38.079303 7926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:38.080598 7926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:38.092300 7931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:38.092778 7932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:38.093051 7934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:38.093742 7926 server_base.cc:1034] running on GCE node
I20260620 15:00:38.094482 7926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:38.095805 7926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:38.097007 7926 hybrid_clock.cc:648] HybridClock initialized: now 1781967638096962 us; error 34 us; skew 500 ppm
I20260620 15:00:38.097332 7926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:38 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967638, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:38.104535 7926 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:00:38.106069 7926 webserver.cc:460] Webserver started at http://127.2.70.62:39591/ using document root <none> and password file <none>
I20260620 15:00:38.106719 7926 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:38.106817 7926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:38.107088 7926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:38.110543 7926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "14cd1ade7857402bbcedd99708349193"
format_stamp: "Formatted at 2026-06-20 15:00:38 on dist-test-slave-n5vv"
server_key: "5a633844ccda847e6de0102e801dd393"
server_key_iv: "f0ebfc7091129a520df9dc971d2274e7"
server_key_version: "encryptionkey@0"
I20260620 15:00:38.111263 7926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "14cd1ade7857402bbcedd99708349193"
format_stamp: "Formatted at 2026-06-20 15:00:38 on dist-test-slave-n5vv"
server_key: "5a633844ccda847e6de0102e801dd393"
server_key_iv: "f0ebfc7091129a520df9dc971d2274e7"
server_key_version: "encryptionkey@0"
I20260620 15:00:38.116279 7926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:00:38.119719 7941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:38.121049 7926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:38.121232 7926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "14cd1ade7857402bbcedd99708349193"
format_stamp: "Formatted at 2026-06-20 15:00:38 on dist-test-slave-n5vv"
server_key: "5a633844ccda847e6de0102e801dd393"
server_key_iv: "f0ebfc7091129a520df9dc971d2274e7"
server_key_version: "encryptionkey@0"
I20260620 15:00:38.121455 7926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:38.133662 7926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:38.137630 7926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:38.137895 7926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:38.156113 7926 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:35347
I20260620 15:00:38.156162 7992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:35347 every 8 connection(s)
I20260620 15:00:38.157467 7926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:38.161069 7993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:38.164372 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 7926
I20260620 15:00:38.164598 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:38.165184 2328 external_mini_cluster.cc:1428] Setting key 7049126ee6f0ae5447ca3a04aa37f9b9
I20260620 15:00:38.173396 7993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193: Bootstrap starting.
I20260620 15:00:38.177179 7993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:38.178556 7993 log.cc:824] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:00:38 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:38.181384 7993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193: No bootstrap required, opened a new log
I20260620 15:00:38.187955 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:38.167080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.2.70.62:35347)
0620 15:00:38.167793 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:38.167803 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:38.167843 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:00:38.169347 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:38.169369 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:38.169831 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:38.170161 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:38.171891 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:38.173352 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:38.175150 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:38.175834 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:38.175995 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:38.180736 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:38.180745 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:38.180758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:38.180794 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:38.183915 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:38.184598 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:38.184610 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:38.184616 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:38.184781 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:38.185144 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:38.185152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:38.185156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:38.185729 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:38.186085 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:38.186435 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 15:00:38.186885 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":336,"threads_started":1}
I20260620 15:00:38.189097 7993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd1ade7857402bbcedd99708349193" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35347 } }
I20260620 15:00:38.189459 7993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:38.189622 7993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14cd1ade7857402bbcedd99708349193, State: Initialized, Role: FOLLOWER
I20260620 15:00:38.190176 7993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd1ade7857402bbcedd99708349193" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35347 } }
I20260620 15:00:38.190420 7993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:38.190572 7993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:38.190769 7993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:38.193113 7993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd1ade7857402bbcedd99708349193" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35347 } }
I20260620 15:00:38.193614 7993 leader_election.cc:302] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14cd1ade7857402bbcedd99708349193; no voters:
I20260620 15:00:38.194149 7993 leader_election.cc:288] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:38.194300 7998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:38.195118 7998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [term 1 LEADER]: Becoming Leader. State: Replica: 14cd1ade7857402bbcedd99708349193, State: Running, Role: LEADER
I20260620 15:00:38.195722 7998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd1ade7857402bbcedd99708349193" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35347 } }
I20260620 15:00:38.196056 7993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:38.198539 8000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14cd1ade7857402bbcedd99708349193. Latest consensus state: current_term: 1 leader_uuid: "14cd1ade7857402bbcedd99708349193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd1ade7857402bbcedd99708349193" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35347 } } }
I20260620 15:00:38.198930 8000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:38.198643 7999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14cd1ade7857402bbcedd99708349193" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14cd1ade7857402bbcedd99708349193" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35347 } } }
I20260620 15:00:38.199069 7999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:38.201894 8005 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:38.206406 8005 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:38.219246 8005 catalog_manager.cc:1347] Generated new cluster ID: db7a55724cbe4003b9e22523c9a7cfe3
I20260620 15:00:38.219384 8005 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:38.237959 8005 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:38.239976 8005 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:38.249060 8005 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14cd1ade7857402bbcedd99708349193: Generated new TSK 0
I20260620 15:00:38.250149 8005 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:00:38.301138 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35347
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:38.635900 8019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:38.636178 8019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:38.636278 8019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:38.643527 8019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:38.643631 8019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:38.643741 8019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:38.653327 8019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:38.654539 8019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:38.655784 8019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:38.668599 8027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:38.668790 8025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:38.668994 8024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:38.669553 8019 server_base.cc:1034] running on GCE node
I20260620 15:00:38.670109 8019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:38.670986 8019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:38.672207 8019 hybrid_clock.cc:648] HybridClock initialized: now 1781967638672129 us; error 56 us; skew 500 ppm
I20260620 15:00:38.672592 8019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:38 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967638, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:38.675712 8019 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:00:38.677158 8019 webserver.cc:460] Webserver started at http://127.2.70.1:44787/ using document root <none> and password file <none>
I20260620 15:00:38.677906 8019 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:38.678047 8019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:38.678350 8019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:38.681217 8019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "e9667563f282425cbb303e40af793a48"
format_stamp: "Formatted at 2026-06-20 15:00:38 on dist-test-slave-n5vv"
server_key: "a9f12b8318fab35b311466b6d6ead587"
server_key_iv: "20e15061989026da76dea7a8f1c887b3"
server_key_version: "encryptionkey@0"
I20260620 15:00:38.682052 8019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "e9667563f282425cbb303e40af793a48"
format_stamp: "Formatted at 2026-06-20 15:00:38 on dist-test-slave-n5vv"
server_key: "a9f12b8318fab35b311466b6d6ead587"
server_key_iv: "20e15061989026da76dea7a8f1c887b3"
server_key_version: "encryptionkey@0"
I20260620 15:00:38.686939 8019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:00:38.690594 8034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:38.691865 8019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:00:38.692050 8019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "e9667563f282425cbb303e40af793a48"
format_stamp: "Formatted at 2026-06-20 15:00:38 on dist-test-slave-n5vv"
server_key: "a9f12b8318fab35b311466b6d6ead587"
server_key_iv: "20e15061989026da76dea7a8f1c887b3"
server_key_version: "encryptionkey@0"
I20260620 15:00:38.692260 8019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:38.705319 8019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:38.709056 8019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:38.709323 8019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:38.710322 8019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:38.711591 8019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:38.711690 8019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:38.711822 8019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:38.711891 8019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:38.744601 8019 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:41307
I20260620 15:00:38.744637 8146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:41307 every 8 connection(s)
I20260620 15:00:38.745985 8019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:38.750820 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 8019
I20260620 15:00:38.751029 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:38.751668 2328 external_mini_cluster.cc:1428] Setting key 83db01a932d099711b3e4c9cfcc0ffad
Jun 20 15:00:38 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967638, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:38.769208 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:38.748553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:53663 (local address 127.2.70.62:35347)
0620 15:00:38.749152 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:38.749156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:38.750189 (+ 1033us) server_negotiation.cc:407] Connection header received
0620 15:00:38.752303 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:38.752307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:38.752370 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:38.752550 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:38.754298 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:38.755004 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:38.757589 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:38.757997 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:38.758094 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:38.762378 (+ 4284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:38.762387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:38.762395 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:38.762448 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:38.765898 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:38.766624 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:38.766633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:38.766639 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:38.766783 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:38.767195 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:38.767203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:38.767208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:38.767603 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:38.767750 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:38.768473 (+ 723us) server_negotiation.cc:299] Negotiation successful
0620 15:00:38.768734 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":388,"threads_started":1}
I20260620 15:00:38.769943 8150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:38.749194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35347 (local address 127.2.70.1:53663)
0620 15:00:38.750046 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:38.750083 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:38.752026 (+ 1943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:38.752750 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:38.752758 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:38.753297 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:38.754129 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:38.754155 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:38.755163 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:38.755172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:38.757405 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:38.757423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:38.758219 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:38.758229 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:38.758563 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:38.759318 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:38.759350 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:38.762131 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:38.766040 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:38.766047 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:38.766061 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:38.766487 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:38.766902 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:38.766906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:38.766908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:38.767083 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:38.767731 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:38.767736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:38.768074 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 15:00:38.768315 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 15:00:38.768751 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":372,"threads_started":1}
I20260620 15:00:38.771498 8147 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35347
I20260620 15:00:38.771840 8147 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:38.772771 8147 heartbeater.cc:511] Master 127.2.70.62:35347 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:00:38.775135 7958 ts_manager.cc:194] Registered new tserver with Master: e9667563f282425cbb303e40af793a48 (127.2.70.1:41307)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
I20260620 15:00:38.776528 7958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:53663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:00:38.800760 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35347
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:39.126889 8153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:39.127197 8153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:39.127359 8153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:39.134415 8153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:39.134572 8153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:39.134749 8153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:39.144222 8153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:39.145551 8153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:39.146929 8153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:39.160918 8161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:39.160878 8159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:39.161224 8158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:39.167038 8153 server_base.cc:1034] running on GCE node
I20260620 15:00:39.167632 8153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:39.168692 8153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:39.169904 8153 hybrid_clock.cc:648] HybridClock initialized: now 1781967639169836 us; error 56 us; skew 500 ppm
I20260620 15:00:39.170228 8153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.173750 8153 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:00:39.175428 8153 webserver.cc:460] Webserver started at http://127.2.70.2:44293/ using document root <none> and password file <none>
I20260620 15:00:39.176400 8153 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:39.176506 8153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:39.176776 8153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:39.179687 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "1714272e170345dbbbe4751ba911e041"
format_stamp: "Formatted at 2026-06-20 15:00:39 on dist-test-slave-n5vv"
server_key: "e4baf31da6739eb936e855b1e3ac0163"
server_key_iv: "ddf4503abedeb9d4799c95ddf0709f3a"
server_key_version: "encryptionkey@0"
I20260620 15:00:39.180382 8153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "1714272e170345dbbbe4751ba911e041"
format_stamp: "Formatted at 2026-06-20 15:00:39 on dist-test-slave-n5vv"
server_key: "e4baf31da6739eb936e855b1e3ac0163"
server_key_iv: "ddf4503abedeb9d4799c95ddf0709f3a"
server_key_version: "encryptionkey@0"
I20260620 15:00:39.185210 8153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:00:39.188544 8168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:39.189972 8153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:39.190140 8153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "1714272e170345dbbbe4751ba911e041"
format_stamp: "Formatted at 2026-06-20 15:00:39 on dist-test-slave-n5vv"
server_key: "e4baf31da6739eb936e855b1e3ac0163"
server_key_iv: "ddf4503abedeb9d4799c95ddf0709f3a"
server_key_version: "encryptionkey@0"
I20260620 15:00:39.190330 8153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:39.206449 8153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:39.210386 8153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:39.210656 8153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:39.211565 8153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:39.212806 8153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:39.212913 8153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:39.213044 8153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:39.213115 8153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:39.245077 8153 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:36715
I20260620 15:00:39.245134 8280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:36715 every 8 connection(s)
I20260620 15:00:39.246454 8153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:39.249917 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 8153
I20260620 15:00:39.250149 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:39.250748 2328 external_mini_cluster.cc:1428] Setting key ce90d9378c59b4931cc27f9bc9862b49
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:39.268151 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.249173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:39309 (local address 127.2.70.62:35347)
0620 15:00:39.249467 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.249510 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.250714 (+ 1204us) server_negotiation.cc:407] Connection header received
0620 15:00:39.252753 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.252758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.252842 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.252946 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.254448 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.255147 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.256719 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.257129 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.257256 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:39.261070 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:39.261079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:39.261087 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:39.261131 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:39.264810 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.265585 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.265594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.265600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.265754 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.266205 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.266213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.266220 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.266606 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:39.266755 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:39.267425 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 15:00:39.267661 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:00:39.268854 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.249738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35347 (local address 127.2.70.2:39309)
0620 15:00:39.250564 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:39.250607 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:39.252483 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:39.253181 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:39.253189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:39.253746 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:39.254295 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:39.254311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.255302 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.255306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:39.256561 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:39.256572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.257345 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.257351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:39.257633 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:39.258345 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:39.258370 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:39.260811 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:39.264969 (+ 4158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:39.264977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:39.264991 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:39.265422 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:39.265863 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:39.265866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:39.265869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:39.266081 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:39.266724 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:39.266730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:39.267065 (+ 335us) client_negotiation.cc:769] Sending connection context
0620 15:00:39.267289 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 15:00:39.267685 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":356,"threads_started":1}
I20260620 15:00:39.270452 8281 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35347
I20260620 15:00:39.270773 8281 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:00:39.271647 8281 heartbeater.cc:511] Master 127.2.70.62:35347 requested a full tablet report, sending...
I20260620 15:00:39.273242 7958 ts_manager.cc:194] Registered new tserver with Master: 1714272e170345dbbbe4751ba911e041 (127.2.70.2:36715)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
I20260620 15:00:39.274300 7958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:39309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:00:39.298275 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35347
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:39.616918 8286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:39.617220 8286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:39.617384 8286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:39.624583 8286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:39.624735 8286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:39.624918 8286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:39.634616 8286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:39.635913 8286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:39.637218 8286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:39.654568 8291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:39.654609 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:39.654659 8292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:39.656023 8286 server_base.cc:1034] running on GCE node
I20260620 15:00:39.657161 8286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:39.658458 8286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:39.659694 8286 hybrid_clock.cc:648] HybridClock initialized: now 1781967639659642 us; error 40 us; skew 500 ppm
I20260620 15:00:39.660094 8286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.664075 8286 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:00:39.666927 8286 webserver.cc:460] Webserver started at http://127.2.70.3:46393/ using document root <none> and password file <none>
I20260620 15:00:39.667847 8286 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:39.667974 8286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:39.668290 8286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:39.672009 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "112cab2825fe4433be251f0df96643a7"
format_stamp: "Formatted at 2026-06-20 15:00:39 on dist-test-slave-n5vv"
server_key: "eb2fc02854ce7696f01a6d23d8c19758"
server_key_iv: "a9b9a558f2604a3a786c176c05381590"
server_key_version: "encryptionkey@0"
I20260620 15:00:39.673053 8286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "112cab2825fe4433be251f0df96643a7"
format_stamp: "Formatted at 2026-06-20 15:00:39 on dist-test-slave-n5vv"
server_key: "eb2fc02854ce7696f01a6d23d8c19758"
server_key_iv: "a9b9a558f2604a3a786c176c05381590"
server_key_version: "encryptionkey@0"
I20260620 15:00:39.680603 8286 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 15:00:39.685693 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:39.687350 8286 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:00:39.687525 8286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "112cab2825fe4433be251f0df96643a7"
format_stamp: "Formatted at 2026-06-20 15:00:39 on dist-test-slave-n5vv"
server_key: "eb2fc02854ce7696f01a6d23d8c19758"
server_key_iv: "a9b9a558f2604a3a786c176c05381590"
server_key_version: "encryptionkey@0"
I20260620 15:00:39.687737 8286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:39.719039 8286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:39.723142 8286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:39.723424 8286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:39.724364 8286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:39.725891 8286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:39.725986 8286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:39.726145 8286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:39.726224 8286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:39.759548 8286 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:41283
I20260620 15:00:39.759591 8413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:41283 every 8 connection(s)
I20260620 15:00:39.760809 8286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:39.767758 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 8286
I20260620 15:00:39.767974 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:39.768361 2328 external_mini_cluster.cc:1428] Setting key c105ea027ee45cbcda304709f2ebbd72
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:39.779335 8147 heartbeater.cc:503] Master 127.2.70.62:35347 was elected leader, sending a full tablet report...
I20260620 15:00:39.780897 8149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.763339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:39019 (local address 127.2.70.62:35347)
0620 15:00:39.763548 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.763555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.764675 (+ 1120us) server_negotiation.cc:407] Connection header received
0620 15:00:39.766468 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.766473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.766538 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.766648 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.768753 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.769795 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.771813 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.772208 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.772299 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:39.775547 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:39.775552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:39.775555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:39.775583 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:39.777975 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.778637 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.778641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.778643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.778757 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.779119 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.779123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.779125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.779453 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:39.779591 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:39.780369 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 15:00:39.780530 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:00:39.782027 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.763852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35347 (local address 127.2.70.3:39019)
0620 15:00:39.764524 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:39.764566 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:39.766184 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:39.766951 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:39.766963 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:39.767794 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:39.768560 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:39.768579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.769942 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.769946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:39.771682 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:39.771694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.772377 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.772382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:39.772609 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:39.773266 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:39.773287 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:39.775376 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:39.778112 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:39.778118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:39.778130 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:39.778546 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:39.778848 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:39.778851 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:39.778853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:39.779021 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:39.779580 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:39.779585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:39.779989 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 15:00:39.780288 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 15:00:39.780838 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":276,"threads_started":1}
I20260620 15:00:39.783433 8414 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35347
I20260620 15:00:39.783679 8414 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:39.784317 8414 heartbeater.cc:511] Master 127.2.70.62:35347 requested a full tablet report, sending...
I20260620 15:00:39.785701 7958 ts_manager.cc:194] Registered new tserver with Master: 112cab2825fe4433be251f0df96643a7 (127.2.70.3:41283)
I20260620 15:00:39.786465 7958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:39019
I20260620 15:00:39.795401 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/test-admin.keytab.
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.822264 2328 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 15:00:39.827275 8421 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:39.824772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.2.70.1:41307)
0620 15:00:39.825627 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.825634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.825660 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:00:39.825725 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.825732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.825954 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.826078 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.826887 (+ 809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33686: BlockingRecv error: recv got EOF from 127.0.0.1:33686 (error 108)
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":391,"threads_started":1}
W20260620 15:00:39.827329 2328 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.828840 2328 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.830245 2328 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:00:39.844754 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.831866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.2.70.1:41307)
0620 15:00:39.832062 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.832070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.832256 (+ 186us) server_negotiation.cc:407] Connection header received
0620 15:00:39.832459 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.832470 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.832534 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.832634 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.833624 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.834604 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.835827 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.836392 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.836547 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:39.839804 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:39.839814 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:39.839828 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:39.839863 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:39.842414 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.842967 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.842973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.842975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.843082 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.843370 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.843373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.843375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.843714 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:39.843913 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:39.844307 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 15:00:39.844508 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:00:39.850642 8421 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:39.848869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.2.70.1:41307)
0620 15:00:39.849073 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.849081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.849353 (+ 272us) server_negotiation.cc:407] Connection header received
0620 15:00:39.849542 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.849551 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.849618 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.849714 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.850401 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33706: BlockingRecv error: recv got EOF from 127.0.0.1:33706 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 15:00:39.850907 2328 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.852418 2328 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.853677 2328 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:00:39.867612 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.854828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.2.70.1:41307)
0620 15:00:39.855014 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.855022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.855297 (+ 275us) server_negotiation.cc:407] Connection header received
0620 15:00:39.855457 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.855462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.855533 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.855624 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.856483 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.857183 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.858531 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.859005 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.859167 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:39.862716 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:39.862720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:39.862723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:39.862756 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:39.865127 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.865802 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.865808 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.865811 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.865931 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.866293 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.866298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.866301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.866579 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:39.866836 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:39.867141 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 15:00:39.867340 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260620 15:00:39.872136 8421 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:39.870274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33722 (local address 127.2.70.1:41307)
0620 15:00:39.870415 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.870419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.870866 (+ 447us) server_negotiation.cc:407] Connection header received
0620 15:00:39.871014 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.871017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.871079 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.871154 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.871879 (+ 725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33722: BlockingRecv error: recv got EOF from 127.0.0.1:33722 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 15:00:39.872448 2328 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781967589245963-2328-0/krb5kdc/krb5cc' of type FILE
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.874069 2328 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:39.875393 2328 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 15:00:39 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:00:39.889708 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.876672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.2.70.1:41307)
0620 15:00:39.876884 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.876890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.877135 (+ 245us) server_negotiation.cc:407] Connection header received
0620 15:00:39.877319 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.877323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.877377 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.877472 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.878498 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.879232 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.880496 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.880943 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.881074 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:39.884687 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:39.884692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:39.884695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:39.884726 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:39.887262 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.887901 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.887906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.887910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.888045 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.888392 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.888397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.888401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.888665 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:39.888980 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:39.889166 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 15:00:39.889367 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:00:39.891270 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 8019
I20260620 15:00:39.903512 8421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:39.892108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.2.70.1:41307)
0620 15:00:39.892263 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:39.892267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:39.892667 (+ 400us) server_negotiation.cc:407] Connection header received
0620 15:00:39.892827 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:39.892831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:39.892905 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:39.892999 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:39.894027 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.894747 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.895934 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:39.896460 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:39.896623 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:39.898684 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:39.898690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:39.898695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:39.898731 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:39.900954 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.901669 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.901676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.901679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.901807 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:39.902161 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:39.902167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:39.902170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:39.902415 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:39.902679 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:39.903001 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 15:00:39.903199 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:00:39.904102 8142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:39.987056 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 8019
I20260620 15:00:40.003618 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 8153
Jun 20 15:00:40 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:00:40.018285 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:40.004500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.2.70.2:36715)
0620 15:00:40.005081 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:40.005088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:40.005113 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:00:40.005177 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:40.005183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:40.005391 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:40.005575 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:40.006539 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:40.007552 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:40.008782 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:40.009319 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:40.009412 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:40.013228 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:40.013239 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:40.013252 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:40.013281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:40.015848 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:40.016415 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:40.016422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:40.016424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:40.016532 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:40.016854 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:40.016857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:40.016859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:40.017170 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:40.017381 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:40.017699 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 15:00:40.017901 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":332,"threads_started":1}
I20260620 15:00:40.019582 8276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:40.101632 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 8153
I20260620 15:00:40.117813 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 8286
Jun 20 15:00:40 dist-test-slave-n5vv krb5kdc[7913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:00:40.133797 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:40.118824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.2.70.3:41283)
0620 15:00:40.119445 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:40.119454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:40.119485 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:00:40.119544 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:40.119550 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:40.119759 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:40.119940 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:40.121040 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:40.122241 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:40.123487 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:40.124045 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:40.124180 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:40.128359 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:40.128371 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:40.128385 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:40.128419 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:40.131285 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:40.131937 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:40.131944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:40.131946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:40.132061 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:40.132418 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:40.132422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:40.132424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:40.132758 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:40.132994 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:40.133275 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 15:00:40.133459 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":332,"threads_started":1}
I20260620 15:00:40.135130 8409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:40.218441 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 8286
I20260620 15:00:40.234058 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 7926
I20260620 15:00:40.235196 7988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:40.335464 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 7926
2026-06-20T15:00:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3904 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:40 dist-test-slave-n5vv krb5kdc[8441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:40 dist-test-slave-n5vv krb5kdc[8441](info): set up 2 sockets
Jun 20 15:00:40 dist-test-slave-n5vv krb5kdc[8441](info): commencing operation
krb5kdc: starting...
W20260620 15:00:41.458292 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:41 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.62.keytab.
Entry for principal oryx/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.62.keytab.
I20260620 15:00:41.668839 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:40617
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.62.keytab
--principal=oryx/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:40617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:41.987353 8454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:41.987636 8454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:41.987811 8454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:41.995074 8454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:41.995209 8454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:41.995287 8454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:41.995378 8454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:41.995452 8454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:42.006120 8454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:40617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.62.keytab
--principal=oryx/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:40617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:42.007521 8454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:42.008862 8454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:42.020283 8462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:42.020388 8459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:42.020355 8460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:42.021246 8454 server_base.cc:1034] running on GCE node
I20260620 15:00:42.022110 8454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:42.023554 8454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:42.024788 8454 hybrid_clock.cc:648] HybridClock initialized: now 1781967642024738 us; error 38 us; skew 500 ppm
I20260620 15:00:42.025146 8454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:42 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:42.028318 8454 init.cc:377] Logged in from keytab as oryx/127.2.70.62@KRBTEST.COM (short username oryx)
I20260620 15:00:42.029918 8454 webserver.cc:460] Webserver started at http://127.2.70.62:41527/ using document root <none> and password file <none>
I20260620 15:00:42.030603 8454 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:42.030778 8454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:42.031114 8454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:42.034133 8454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "3e355101e7d7451493225e2674944836"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-n5vv"
server_key: "1c09362f037ee818a84c483e75b05e6d"
server_key_iv: "22a9396348d65383f1390533ec51f108"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.034888 8454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "3e355101e7d7451493225e2674944836"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-n5vv"
server_key: "1c09362f037ee818a84c483e75b05e6d"
server_key_iv: "22a9396348d65383f1390533ec51f108"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.040113 8454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:42.043587 8469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:42.044903 8454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:42.045115 8454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "3e355101e7d7451493225e2674944836"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-n5vv"
server_key: "1c09362f037ee818a84c483e75b05e6d"
server_key_iv: "22a9396348d65383f1390533ec51f108"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.045320 8454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:42.066941 8454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:42.070879 8454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:42.071170 8454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:42.089365 8454 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:40617
I20260620 15:00:42.089395 8520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:40617 every 8 connection(s)
I20260620 15:00:42.090740 8454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:42.094424 8521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:42.098379 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 8454
I20260620 15:00:42.098619 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:42.099392 2328 external_mini_cluster.cc:1428] Setting key 36231c052954c232826662145f9a7447
I20260620 15:00:42.107918 8521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: Bootstrap starting.
I20260620 15:00:42.111752 8521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:42.113113 8521 log.cc:824] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:00:42 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:42.116235 8521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: No bootstrap required, opened a new log
I20260620 15:00:42.122306 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:42.101381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.2.70.62:40617)
0620 15:00:42.102223 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:42.102234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:42.102272 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:00:42.103763 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:42.103785 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:42.104201 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:42.104525 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:42.106417 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.107767 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.109403 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.110164 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.110298 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:42.115059 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:42.115067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:42.115082 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:42.115119 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:42.118345 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:42.119058 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:42.119064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:42.119070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:42.119221 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:42.119598 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:42.119603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:42.119610 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:42.120106 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:42.120430 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:42.120817 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 15:00:42.121239 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":345,"threads_started":1}
I20260620 15:00:42.124034 8521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } }
I20260620 15:00:42.124397 8521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:42.124518 8521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e355101e7d7451493225e2674944836, State: Initialized, Role: FOLLOWER
I20260620 15:00:42.125151 8521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } }
I20260620 15:00:42.125391 8521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:42.125551 8521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:42.125736 8521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:42.127821 8521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } }
I20260620 15:00:42.128289 8521 leader_election.cc:302] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e355101e7d7451493225e2674944836; no voters:
I20260620 15:00:42.128934 8521 leader_election.cc:288] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:42.129081 8526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:42.130126 8526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 1 LEADER]: Becoming Leader. State: Replica: 3e355101e7d7451493225e2674944836, State: Running, Role: LEADER
I20260620 15:00:42.130777 8526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } }
I20260620 15:00:42.131347 8521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:42.133692 8528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e355101e7d7451493225e2674944836. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } }
I20260620 15:00:42.133720 8527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } }
I20260620 15:00:42.134063 8528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:42.134080 8527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:42.135118 8533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:42.140003 8533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:42.148806 8533 catalog_manager.cc:1347] Generated new cluster ID: bf8a40f7f97a4ad88e5e36747ef37dc7
I20260620 15:00:42.148931 8533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:42.155665 8533 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:42.156881 8533 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:42.170737 8533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: Generated new TSK 0
I20260620 15:00:42.171618 8533 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab.
Entry for principal oryx/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab.
I20260620 15:00:42.220860 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:40617
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab
--principal=oryx/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:42.551585 8547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:42.551851 8547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:42.551941 8547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:42.559080 8547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:42.559244 8547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:42.559381 8547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:42.568722 8547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab
--principal=oryx/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:40617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:42.570026 8547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:42.571254 8547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:42.585158 8552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:42.585270 8555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:42.585259 8553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:42.586648 8547 server_base.cc:1034] running on GCE node
I20260620 15:00:42.587325 8547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:42.588335 8547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:42.589593 8547 hybrid_clock.cc:648] HybridClock initialized: now 1781967642589527 us; error 53 us; skew 500 ppm
I20260620 15:00:42.589941 8547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:42 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:42.593034 8547 init.cc:377] Logged in from keytab as oryx/127.2.70.1@KRBTEST.COM (short username oryx)
I20260620 15:00:42.594520 8547 webserver.cc:460] Webserver started at http://127.2.70.1:37371/ using document root <none> and password file <none>
I20260620 15:00:42.595228 8547 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:42.595367 8547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:42.595670 8547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:42.598762 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "1927c10bb1804fbda5f90a3759744e64"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-n5vv"
server_key: "52bd33cb843c6281135e5e8b343b5781"
server_key_iv: "59cbc2e9317bdffd85af7229748fe675"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.599545 8547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "1927c10bb1804fbda5f90a3759744e64"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-n5vv"
server_key: "52bd33cb843c6281135e5e8b343b5781"
server_key_iv: "59cbc2e9317bdffd85af7229748fe675"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.604696 8547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260620 15:00:42.608183 8562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:42.609634 8547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:42.609795 8547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "1927c10bb1804fbda5f90a3759744e64"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-n5vv"
server_key: "52bd33cb843c6281135e5e8b343b5781"
server_key_iv: "59cbc2e9317bdffd85af7229748fe675"
server_key_version: "encryptionkey@0"
I20260620 15:00:42.609995 8547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:42.637974 8547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:42.641973 8547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:42.642284 8547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:42.643260 8547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:42.644569 8547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:42.644666 8547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:42.644800 8547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:42.644867 8547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:42.678020 8547 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:40921
I20260620 15:00:42.678068 8674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:40921 every 8 connection(s)
I20260620 15:00:42.679262 8547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:42.681427 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 8547
I20260620 15:00:42.681700 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:42.682404 2328 external_mini_cluster.cc:1428] Setting key 789719e1ae1648ab397474a11e117dab
Jun 20 15:00:42 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:42.701850 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:42.681949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:36959 (local address 127.2.70.62:40617)
0620 15:00:42.682463 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:42.682471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:42.683588 (+ 1117us) server_negotiation.cc:407] Connection header received
0620 15:00:42.686044 (+ 2456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:42.686049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:42.686136 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:42.686233 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:42.688402 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.689574 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.692117 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.692524 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.692673 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:42.696041 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:42.696046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:42.696049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:42.696082 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:42.698647 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:42.699345 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:42.699351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:42.699355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:42.699471 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:42.700039 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:42.700043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:42.700045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:42.700288 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:42.700394 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:42.701364 (+ 970us) server_negotiation.cc:299] Negotiation successful
0620 15:00:42.701598 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":239,"threads_started":1}
I20260620 15:00:42.703481 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:42.682617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.1:36959)
0620 15:00:42.683408 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:42.683463 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:42.685721 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:42.686528 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:42.686542 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:42.687349 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:42.688185 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:42.688213 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.689741 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.689750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:42.691959 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:42.691971 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:42.692750 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:42.692759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:42.693008 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:42.693693 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:42.693716 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:42.695825 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:42.698782 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:42.698791 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:42.698808 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:42.699226 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:42.699616 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:42.699622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:42.699629 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:42.699913 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:42.700424 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:42.700434 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:42.700933 (+ 499us) client_negotiation.cc:769] Sending connection context
0620 15:00:42.701294 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 15:00:42.701950 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":307,"threads_started":1}
WARNING: no policy specified for oryx/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:00:42.705086 8675 heartbeater.cc:347] Connected to a master server at 127.2.70.62:40617
I20260620 15:00:42.705394 8675 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:42.706157 8675 heartbeater.cc:511] Master 127.2.70.62:40617 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.2@KRBTEST.COM" created.
I20260620 15:00:42.708510 8486 ts_manager.cc:194] Registered new tserver with Master: 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921)
I20260620 15:00:42.710635 8486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:36959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab.
Entry for principal oryx/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab.
I20260620 15:00:42.732502 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:40617
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab
--principal=oryx/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:43.052043 8681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:43.052477 8681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:43.052647 8681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:43.060086 8681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:43.060200 8681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:43.060312 8681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:43.069794 8681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab
--principal=oryx/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:40617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:43.071020 8681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:43.072222 8681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:43.085186 8686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:43.085214 8689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:43.085258 8687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:43.086171 8681 server_base.cc:1034] running on GCE node
I20260620 15:00:43.086859 8681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:43.087929 8681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:43.089177 8681 hybrid_clock.cc:648] HybridClock initialized: now 1781967643089130 us; error 47 us; skew 500 ppm
I20260620 15:00:43.089653 8681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:43 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967643, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:43.093123 8681 init.cc:377] Logged in from keytab as oryx/127.2.70.2@KRBTEST.COM (short username oryx)
I20260620 15:00:43.094780 8681 webserver.cc:460] Webserver started at http://127.2.70.2:45885/ using document root <none> and password file <none>
I20260620 15:00:43.095538 8681 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:43.095690 8681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:43.096011 8681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:43.099290 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "7c6f288b292f48d9a59f76e4d97503c0"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-n5vv"
server_key: "228acc7d36a092fcb4da62078bd1745c"
server_key_iv: "83416259afcc94b231ad76526037fe97"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.100112 8681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "7c6f288b292f48d9a59f76e4d97503c0"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-n5vv"
server_key: "228acc7d36a092fcb4da62078bd1745c"
server_key_iv: "83416259afcc94b231ad76526037fe97"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.106079 8681 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260620 15:00:43.109625 8696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.111153 8681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:43.111338 8681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "7c6f288b292f48d9a59f76e4d97503c0"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-n5vv"
server_key: "228acc7d36a092fcb4da62078bd1745c"
server_key_iv: "83416259afcc94b231ad76526037fe97"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.111610 8681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:43.129407 8681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:43.133926 8681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:43.134240 8681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:43.135224 8681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:43.136653 8681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:43.136759 8681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.136919 8681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:43.137006 8681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.171818 8681 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:36669
I20260620 15:00:43.171861 8808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:36669 every 8 connection(s)
I20260620 15:00:43.173180 8681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:43.182482 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 8681
I20260620 15:00:43.182695 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:43.183189 2328 external_mini_cluster.cc:1428] Setting key 08a0e6571c8ab8d69ef0482da1fb5e76
Jun 20 15:00:43 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967643, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:43.194095 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.175892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:38485 (local address 127.2.70.62:40617)
0620 15:00:43.176056 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.176060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.177540 (+ 1480us) server_negotiation.cc:407] Connection header received
0620 15:00:43.179366 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.179372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.179457 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.179562 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:43.181021 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.181815 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.183486 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.184048 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.184173 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.188308 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:43.188313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:43.188316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:43.188356 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:43.191033 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.191773 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.191783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.191786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.191908 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.192329 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.192348 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.192352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.192616 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:43.192775 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.193513 (+ 738us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.193744 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:00:43.195137 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.176321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.2:38485)
0620 15:00:43.177274 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.177312 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.179091 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.179786 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.179794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.180286 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:43.180852 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.180868 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.181972 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.181977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.183346 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.183358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.184277 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.184284 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.184506 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.185226 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:43.185261 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:43.188084 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:43.191197 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.191206 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.191224 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.191645 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.192014 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.192023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.192026 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.192219 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.192752 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:43.192760 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:43.193125 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.193408 (+ 283us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.193998 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":294,"threads_started":1}
I20260620 15:00:43.196735 8809 heartbeater.cc:347] Connected to a master server at 127.2.70.62:40617
I20260620 15:00:43.197057 8809 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:43.197845 8809 heartbeater.cc:511] Master 127.2.70.62:40617 requested a full tablet report, sending...
I20260620 15:00:43.199409 8486 ts_manager.cc:194] Registered new tserver with Master: 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2:36669)
I20260620 15:00:43.200253 8486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:38485
WARNING: no policy specified for oryx/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab.
Entry for principal oryx/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab.
I20260620 15:00:43.233361 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:40617
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab
--principal=oryx/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:43.559526 8814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:43.559810 8814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:43.559980 8814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:43.568140 8814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:43.568276 8814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:43.568532 8814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:43.579767 8814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab
--principal=oryx/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:40617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:43.581197 8814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:43.582592 8814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:43.595832 8819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:43.595953 8822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:43.595974 8820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:43.597004 8814 server_base.cc:1034] running on GCE node
I20260620 15:00:43.597821 8814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:43.598995 8814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:43.600251 8814 hybrid_clock.cc:648] HybridClock initialized: now 1781967643600187 us; error 56 us; skew 500 ppm
I20260620 15:00:43.600585 8814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:43 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967643, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:43.603991 8814 init.cc:377] Logged in from keytab as oryx/127.2.70.3@KRBTEST.COM (short username oryx)
I20260620 15:00:43.605743 8814 webserver.cc:460] Webserver started at http://127.2.70.3:36285/ using document root <none> and password file <none>
I20260620 15:00:43.606536 8814 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:43.606643 8814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:43.606909 8814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:43.610247 8814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "e48b32dc05114c65889487a5089f96d3"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-n5vv"
server_key: "809a7bbacdd033e78fab13f236aec82f"
server_key_iv: "9e799a7cb00fb4f4af32138842640593"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.611001 8814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "e48b32dc05114c65889487a5089f96d3"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-n5vv"
server_key: "809a7bbacdd033e78fab13f236aec82f"
server_key_iv: "9e799a7cb00fb4f4af32138842640593"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.616636 8814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:43.620586 8829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.622216 8814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:43.622392 8814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "e48b32dc05114c65889487a5089f96d3"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-n5vv"
server_key: "809a7bbacdd033e78fab13f236aec82f"
server_key_iv: "9e799a7cb00fb4f4af32138842640593"
server_key_version: "encryptionkey@0"
I20260620 15:00:43.622630 8814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:43.641816 8814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:43.646683 8814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:43.647030 8814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:43.648118 8814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:43.649693 8814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:43.649771 8814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.649967 8814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:43.650030 8814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:43.686664 8814 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:39307
I20260620 15:00:43.686693 8941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:39307 every 8 connection(s)
I20260620 15:00:43.688011 8814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:43.693951 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 8814
I20260620 15:00:43.694204 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:43.694772 2328 external_mini_cluster.cc:1428] Setting key aab05190e7fa19cda58139d81c84e205
Jun 20 15:00:43 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967643, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:43.708886 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.690603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:49365 (local address 127.2.70.62:40617)
0620 15:00:43.690778 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:43.690782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:43.692044 (+ 1262us) server_negotiation.cc:407] Connection header received
0620 15:00:43.693996 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:43.694000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:43.694124 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:43.694228 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:43.696371 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.697344 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.699165 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.699637 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.699773 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.703201 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:43.703206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:43.703209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:43.703240 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:43.705852 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.706551 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.706559 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.706567 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.706683 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:43.707110 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:43.707115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:43.707118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:43.707396 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:43.707559 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:43.708296 (+ 737us) server_negotiation.cc:299] Negotiation successful
0620 15:00:43.708524 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:00:43.709921 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:43.691166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.3:49365)
0620 15:00:43.691868 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:43.691906 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:43.693634 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:43.694508 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:43.694521 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:43.695302 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:43.696178 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.696206 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.697563 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.697569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.699020 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:43.699031 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:43.699873 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:43.699881 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:43.700126 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:43.700796 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:43.700822 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:43.702986 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:43.705991 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.706001 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.706017 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.706425 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.706791 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:43.706797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:43.706800 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:43.706992 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:43.707524 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:43.707532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:43.707891 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 15:00:43.708168 (+ 277us) client_negotiation.cc:240] Negotiation successful
0620 15:00:43.708696 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":289,"threads_started":1}
I20260620 15:00:43.711215 8942 heartbeater.cc:347] Connected to a master server at 127.2.70.62:40617
I20260620 15:00:43.711539 8942 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:43.712249 8942 heartbeater.cc:511] Master 127.2.70.62:40617 requested a full tablet report, sending...
I20260620 15:00:43.713794 8486 ts_manager.cc:194] Registered new tserver with Master: e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307)
I20260620 15:00:43.714273 8675 heartbeater.cc:503] Master 127.2.70.62:40617 was elected leader, sending a full tablet report...
I20260620 15:00:43.714758 8486 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:49365
I20260620 15:00:43.723428 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.70.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.61.keytab.
Entry for principal oryx/127.2.70.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.61.keytab.
I20260620 15:00:43.773851 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.61:36919
--webserver_interface=127.2.70.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.61.keytab
--principal=oryx/127.2.70.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:40617,127.2.70.61:36919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:44.104754 8947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:44.105049 8947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:44.105144 8947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:44.113220 8947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:44.113356 8947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:44.113436 8947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:44.113544 8947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:44.113617 8947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:44.123993 8947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:40617,127.2.70.61:36919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.61.keytab
--principal=oryx/127.2.70.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.61:36919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.70.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
W20260620 15:00:44.124333 8947 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.70.62:40617,127.2.70.61:36919'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 15:00:44.125330 8947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:44.126751 8947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:44.138991 8952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:44.138972 8955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:44.139050 8953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:44.139961 8947 server_base.cc:1034] running on GCE node
I20260620 15:00:44.140792 8947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:44.142259 8947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:44.143465 8947 hybrid_clock.cc:648] HybridClock initialized: now 1781967644143405 us; error 38 us; skew 500 ppm
I20260620 15:00:44.143831 8947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967644, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:44.146943 8947 init.cc:377] Logged in from keytab as oryx/127.2.70.61@KRBTEST.COM (short username oryx)
I20260620 15:00:44.148422 8947 webserver.cc:460] Webserver started at http://127.2.70.61:37765/ using document root <none> and password file <none>
I20260620 15:00:44.149149 8947 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:44.149261 8947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:44.149690 8947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:44.152618 8947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data/instance:
uuid: "4d3023623215480aa17e5317c6334491"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "695d0e820a861f47abf8ee56282102a5"
server_key_iv: "0cdb03a131459183319f6d3998c97cfe"
server_key_version: "encryptionkey@0"
I20260620 15:00:44.153538 8947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal/instance:
uuid: "4d3023623215480aa17e5317c6334491"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "695d0e820a861f47abf8ee56282102a5"
server_key_iv: "0cdb03a131459183319f6d3998c97cfe"
server_key_version: "encryptionkey@0"
I20260620 15:00:44.158620 8947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 15:00:44.161899 8962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:44.163242 8947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:00:44.163424 8947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
uuid: "4d3023623215480aa17e5317c6334491"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "695d0e820a861f47abf8ee56282102a5"
server_key_iv: "0cdb03a131459183319f6d3998c97cfe"
server_key_version: "encryptionkey@0"
I20260620 15:00:44.163637 8947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:44.185998 8947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:44.190096 8947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:44.190348 8947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:44.202944 8809 heartbeater.cc:503] Master 127.2.70.62:40617 was elected leader, sending a full tablet report...
I20260620 15:00:44.210865 9013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.61:36919 every 8 connection(s)
I20260620 15:00:44.210897 8947 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.61:36919
I20260620 15:00:44.212239 8947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
I20260620 15:00:44.214509 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 8947
I20260620 15:00:44.216823 9014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:44.226068 9014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.61@KRBTEST.COM
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967644, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.61@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:44.239820 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.216877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.2.70.61:36919)
0620 15:00:44.217792 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.217807 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.218184 (+ 377us) server_negotiation.cc:407] Connection header received
0620 15:00:44.219940 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.219961 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.220335 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.220655 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.222464 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.223768 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.225477 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.226237 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.226385 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.231593 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.231602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.231613 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.231651 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.235175 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.236038 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.236044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.236075 (+ 31us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.236280 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.236755 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.236759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.236761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.237293 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.237589 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.238120 (+ 531us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.238554 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":368,"threads_started":1}
I20260620 15:00:44.244851 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.229045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45404)
0620 15:00:44.229783 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.229822 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.229962 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.230802 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.230807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.231406 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.231776 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.231785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.233265 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.233269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.234559 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.234570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.235557 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.235563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.235758 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.236469 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.236497 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.239006 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.242183 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.242192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.242198 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.242733 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.243311 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.243314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.243316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.243425 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.244025 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.244031 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.244169 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.244318 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.244580 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":299,"threads_started":1}
I20260620 15:00:44.244869 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.229102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.2.70.62:40617)
0620 15:00:44.230299 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.230303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.230317 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:00:44.230389 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.230394 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.230605 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.230741 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.231930 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.233037 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.234706 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.235383 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.235484 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.239190 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.239195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.239198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.239230 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.242067 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.243083 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.243087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.243090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.243208 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.243593 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.243596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.243599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.243921 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.244038 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.244451 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.244613 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1108,"thread_start_us":1028,"threads_started":1}
I20260620 15:00:44.246549 9014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967644, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.61@KRBTEST.COM for oryx/127.2.70.61@KRBTEST.COM
I20260620 15:00:44.258278 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.247245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.0.0.1:34194)
0620 15:00:44.247458 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.247478 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.247574 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.247886 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.247889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.248211 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.248334 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.248341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.249313 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.249317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.250313 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.250324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.250999 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.251003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.251145 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.251736 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.251752 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.253705 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.256254 (+ 2549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.256258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.256260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.256636 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.256960 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.256963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.256966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.257080 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.257573 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.257576 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.257690 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.257800 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.258005 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:00:44.258443 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.247514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.2.70.61:36919)
0620 15:00:44.247646 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.247651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.247664 (+ 13us) server_negotiation.cc:407] Connection header received
0620 15:00:44.247700 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.247703 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.247750 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.247824 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.248456 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.249213 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.250432 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.250870 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.250975 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.253856 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.253861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.253865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.253890 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.256139 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.256735 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.256738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.256740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.256860 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.257168 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.257172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.257173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.257418 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.257607 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.257877 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.258119 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":50}
I20260620 15:00:44.263231 9014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Bootstrap starting.
I20260620 15:00:44.266330 9014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:44.267335 9014 log.cc:824] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:44.269995 9014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: No bootstrap required, opened a new log
I20260620 15:00:44.276736 9014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } }
I20260620 15:00:44.277029 9014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:44.277098 9014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3023623215480aa17e5317c6334491, State: Initialized, Role: FOLLOWER
I20260620 15:00:44.277684 9014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } }
I20260620 15:00:44.278451 9021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } } }
I20260620 15:00:44.278781 9021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:44.279500 9014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:00:44.288168 9032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:00:44.288306 9032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:00:44.290182 8947 master_runner.cc:428] Detected that this master 4d3023623215480aa17e5317c6334491 is joining an existing cluster
I20260620 15:00:44.290285 8947 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.70.61:36919
I20260620 15:00:44.312551 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.294145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.2.70.62:40617)
0620 15:00:44.294349 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.294358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.295062 (+ 704us) server_negotiation.cc:407] Connection header received
0620 15:00:44.295568 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.295574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.295662 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.295743 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.297395 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.298574 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.301110 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.301935 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.302059 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.304942 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.304948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.304952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.304992 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.309366 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.310170 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.310177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.310181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.310347 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.310786 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.310793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.310798 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.311143 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.311256 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.311934 (+ 678us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.312173 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:00:44.314201 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.294177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45414)
0620 15:00:44.294922 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.294944 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.295386 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.295987 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.295995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.296741 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.297198 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.297213 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.298752 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.298759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.300943 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.300961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.302398 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.302408 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.302620 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.303357 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.303386 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.304674 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.309538 (+ 4864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.309546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.309552 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.310022 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.310483 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.310487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.310490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.310659 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.311367 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.311373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.311616 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.312815 (+ 1199us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.313154 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":413,"threads_started":1}
I20260620 15:00:44.319924 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.295131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.2.70.61:36919)
0620 15:00:44.295315 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.295325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.305202 (+ 9877us) server_negotiation.cc:407] Connection header received
0620 15:00:44.305365 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.305370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.305433 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.305530 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.306937 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.307631 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.309613 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.310340 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.310438 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.313094 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.313103 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.313108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.313141 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.316808 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.317616 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.317623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.317627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.317791 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.318293 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.318303 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.318309 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.318671 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.318826 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.319299 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.319519 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 15:00:44.321216 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.295466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.0.0.1:34204)
0620 15:00:44.305078 (+ 9612us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.305099 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.305229 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.305736 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.305739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.306419 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.306780 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.306797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.307810 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.307818 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.309450 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.309462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.310736 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.310745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.310939 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.311548 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.311575 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.312583 (+ 1008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.316962 (+ 4379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.316966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.316970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.317473 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.317986 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.317991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.317994 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.318174 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.318906 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.318910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.319048 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.320450 (+ 1402us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.320752 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9473,"thread_start_us":9261,"threads_started":1}
WARNING: no policy specified for oryx/127.2.70.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.60@KRBTEST.COM" created.
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.62@KRBTEST.COM for oryx/127.2.70.61@KRBTEST.COM
I20260620 15:00:44.339066 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.321892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34206 (local address 127.2.70.61:36919)
0620 15:00:44.322063 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.322071 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.323433 (+ 1362us) server_negotiation.cc:407] Connection header received
0620 15:00:44.323773 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.323777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.323836 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.323922 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.325361 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.326358 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.327989 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.328457 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.328599 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.331940 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.331946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.331949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.331977 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.335750 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.336661 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.336667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.336671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.336833 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.337286 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.337291 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.337294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.337705 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.337849 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.338442 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.338668 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:00:44.339277 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.321528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.0.0.1:34206)
0620 15:00:44.323303 (+ 1775us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.323346 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.323576 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.324018 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.324026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.324864 (+ 838us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.325160 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.325174 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.326534 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.326539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.327846 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.327859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.328624 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.328631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.328820 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.329573 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.329614 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.331712 (+ 2098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.335903 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.335918 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.335930 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.336533 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.336963 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.336970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.336977 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.337171 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.337844 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.337855 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.338088 (+ 233us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.338333 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.338719 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"thread_start_us":1391,"threads_started":1}
I20260620 15:00:44.341408 8485 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.70.61:36919: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4d3023623215480aa17e5317c6334491"
member_type: NON_VOTER
last_known_addr {
host: "127.2.70.61"
port: 36919
}
attrs {
promote: true
}
}
dest_uuid: "3e355101e7d7451493225e2674944836"
cas_config_opid_index: -1
I20260620 15:00:44.342614 8485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } }
I20260620 15:00:44.349032 9045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4d3023623215480aa17e5317c6334491 (127.2.70.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } }
I20260620 15:00:44.350292 8989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:44.352852 9045 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.70.61:36919
I20260620 15:00:44.353624 8947 master.cc:537] Master@127.2.70.61:36919 shutting down...
I20260620 15:00:44.355093 8989 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Refusing update from remote peer 3e355101e7d7451493225e2674944836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.60.keytab.
Entry for principal oryx/127.2.70.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.60.keytab.
I20260620 15:00:44.361573 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.60:46775
--webserver_interface=127.2.70.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.60.keytab
--principal=oryx/127.2.70.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:40617,127.2.70.61:36919,127.2.70.60:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:44.369809 9045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:44.370749 9044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } }
I20260620 15:00:44.371140 9044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:44.371345 8947 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:44.371819 8947 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:44.372040 8947 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: stopping tablet replica
W20260620 15:00:44.375792 8965 connection.cc:441] server connection from 127.0.0.1:34206 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34206 (request call id 2) could send its response
W20260620 15:00:44.377142 8473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.61:36919 (error 108)
I20260620 15:00:44.380002 8947 master.cc:552] Master@127.2.70.61:36919 shutdown complete.
I20260620 15:00:44.380084 8947 master_runner.cc:305] Clearing existing system tablet
I20260620 15:00:44.380523 8947 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:44.380625 8947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:44.386283 9041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.385803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.0.0.1:34222)
0620 15:00:44.385985 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.386106 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:36919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:00:44.386713 8473 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 -> Peer 4d3023623215480aa17e5317c6334491 (127.2.70.61:36919): Couldn't send request to peer 4d3023623215480aa17e5317c6334491. Status: Network error: Client connection negotiation failed: client connection to 127.2.70.61:36919: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:00:44.387777 8947 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.001s
I20260620 15:00:44.389667 9051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:44.390117 8947 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:44.390448 8947 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 15:00:44.390542 8947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
uuid: "4d3023623215480aa17e5317c6334491"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "695d0e820a861f47abf8ee56282102a5"
server_key_iv: "0cdb03a131459183319f6d3998c97cfe"
server_key_version: "encryptionkey@0"
I20260620 15:00:44.396265 8947 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:44.403301 8947 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 15:00:44.403484 8947 log.cc:1196] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 15:00:44.403913 8947 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Deleting consensus metadata
I20260620 15:00:44.404171 8947 master_runner.cc:315] Copying system tablet from 127.2.70.62:40617
I20260620 15:00:44.406760 8947 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: tablet copy: Beginning tablet copy session from remote peer at address 127.2.70.62:40617
I20260620 15:00:44.422492 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.407732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.2.70.62:40617)
0620 15:00:44.407911 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.407919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.408515 (+ 596us) server_negotiation.cc:407] Connection header received
0620 15:00:44.408691 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.408697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.408793 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.408904 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.409901 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.411076 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.412799 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.413424 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.413603 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.415910 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.415916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.415922 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.416001 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.419482 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.420256 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.420261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.420264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.420413 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.420813 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.420818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.420821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.421183 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.421347 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.421856 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.422112 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:00:44.422515 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.407595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45416)
0620 15:00:44.408372 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.408402 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.408558 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.408954 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.408960 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.409439 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.409737 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.409748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.411224 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.411230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.412613 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.412627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.413624 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.413631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.413796 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.414516 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.414538 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.415708 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.419627 (+ 3919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.419632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.419637 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.420128 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.420529 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.420534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.420538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.420700 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.421309 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.421315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.421545 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.421734 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.422054 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":496,"threads_started":1}
I20260620 15:00:44.423695 8506 tablet_copy_service.cc:140] P 3e355101e7d7451493225e2674944836: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4d3023623215480aa17e5317c6334491 ({username='oryx', principal='oryx/127.2.70.61@KRBTEST.COM'} at 127.0.0.1:45416)
I20260620 15:00:44.424000 8506 tablet_copy_service.cc:161] P 3e355101e7d7451493225e2674944836: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4d3023623215480aa17e5317c6334491 at {username='oryx', principal='oryx/127.2.70.61@KRBTEST.COM'} at 127.0.0.1:45416: session id = 4d3023623215480aa17e5317c6334491-00000000000000000000000000000000
I20260620 15:00:44.428802 8506 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 15:00:44.431181 8947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 15:00:44.438117 8947 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: tablet copy: Starting download of 0 data blocks...
I20260620 15:00:44.438422 8947 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: tablet copy: Starting download of 1 WAL segments...
I20260620 15:00:44.441159 8947 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 15:00:44.445621 8506 tablet_copy_service.cc:342] P 3e355101e7d7451493225e2674944836: Request end of tablet copy session 4d3023623215480aa17e5317c6334491-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.70.61@KRBTEST.COM'} at 127.0.0.1:45416
I20260620 15:00:44.445842 8506 tablet_copy_service.cc:433] P 3e355101e7d7451493225e2674944836: ending tablet copy session 4d3023623215480aa17e5317c6334491-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4d3023623215480aa17e5317c6334491
I20260620 15:00:44.458346 8947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:44.461099 9058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:44.467135 9059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:44.473907 9061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:44.474718 8947 server_base.cc:1034] running on GCE node
I20260620 15:00:44.475170 8947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:44.475847 8947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:44.477075 8947 hybrid_clock.cc:648] HybridClock initialized: now 1781967644476987 us; error 75 us; skew 500 ppm
I20260620 15:00:44.477301 8947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967644, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:44.479490 8947 init.cc:377] Logged in from keytab as oryx/127.2.70.61@KRBTEST.COM (short username oryx)
I20260620 15:00:44.480876 8947 webserver.cc:460] Webserver started at http://127.2.70.61:36133/ using document root <none> and password file <none>
I20260620 15:00:44.481228 8947 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:44.481313 8947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:44.484539 8947 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 15:00:44.486713 9068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:44.487164 8947 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 15:00:44.487300 8947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
uuid: "4d3023623215480aa17e5317c6334491"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "695d0e820a861f47abf8ee56282102a5"
server_key_iv: "0cdb03a131459183319f6d3998c97cfe"
server_key_version: "encryptionkey@0"
I20260620 15:00:44.487469 8947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:44.505185 8947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:44.509404 8947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:44.558724 8947 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.61:36919
I20260620 15:00:44.558753 9119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.61:36919 every 8 connection(s)
I20260620 15:00:44.559721 8947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
I20260620 15:00:44.564999 9120 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:00:44.568874 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Bootstrap starting.
I20260620 15:00:44.586290 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:00:44.586839 9120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Bootstrap complete.
I20260620 15:00:44.587874 9120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } }
I20260620 15:00:44.588121 9120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4d3023623215480aa17e5317c6334491, State: Initialized, Role: LEARNER
I20260620 15:00:44.588483 9120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } }
I20260620 15:00:44.589697 9120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:44.589607 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } }
I20260620 15:00:44.590083 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: This master's current role is: LEARNER
I20260620 15:00:44.604001 9134 catalog_manager.cc:1259] Loaded cluster ID: bf8a40f7f97a4ad88e5e36747ef37dc7
I20260620 15:00:44.604187 9134 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: loading cluster ID for follower catalog manager: success
I20260620 15:00:44.607761 9134 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: acquiring CA information for follower catalog manager: success
I20260620 15:00:44.610530 9134 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:00:44.717630 8942 heartbeater.cc:503] Master 127.2.70.62:40617 was elected leader, sending a full tablet report...
W20260620 15:00:44.786130 9047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:44.786393 9047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:44.786479 9047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:44.793871 9047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:44.793978 9047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:44.794041 9047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:44.794097 9047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:44.794140 9047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:44.804471 9047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:40617,127.2.70.61:36919,127.2.70.60:46775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.60.keytab
--principal=oryx/127.2.70.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.60:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.70.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:44.805812 9047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:44.807063 9047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:44.819188 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:44.819182 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:44.819182 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:44.820245 9047 server_base.cc:1034] running on GCE node
I20260620 15:00:44.821022 9047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:44.822526 9047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:44.823760 9047 hybrid_clock.cc:648] HybridClock initialized: now 1781967644823704 us; error 56 us; skew 500 ppm
I20260620 15:00:44.824108 9047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967644, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:44.827276 9047 init.cc:377] Logged in from keytab as oryx/127.2.70.60@KRBTEST.COM (short username oryx)
I20260620 15:00:44.828796 9047 webserver.cc:460] Webserver started at http://127.2.70.60:46329/ using document root <none> and password file <none>
I20260620 15:00:44.829469 9047 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:44.829668 9047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:44.829983 9047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:44.832935 9047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data/instance:
uuid: "0caab6fcb3b8497d918f6eee38ac8305"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "b4cae699e615e74bb91d1d9b1c2a9745"
server_key_iv: "1d6ca5f22e50db0fe7cf635c813f1a63"
server_key_version: "encryptionkey@0"
I20260620 15:00:44.833770 9047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal/instance:
uuid: "0caab6fcb3b8497d918f6eee38ac8305"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "b4cae699e615e74bb91d1d9b1c2a9745"
server_key_iv: "1d6ca5f22e50db0fe7cf635c813f1a63"
server_key_version: "encryptionkey@0"
I20260620 15:00:44.838739 9047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:00:44.842659 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:44.844099 9047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:44.844282 9047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
uuid: "0caab6fcb3b8497d918f6eee38ac8305"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "b4cae699e615e74bb91d1d9b1c2a9745"
server_key_iv: "1d6ca5f22e50db0fe7cf635c813f1a63"
server_key_version: "encryptionkey@0"
I20260620 15:00:44.844484 9047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:44.865339 9047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:44.869256 9047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:44.869604 9047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:44.888986 9047 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.60:46775
I20260620 15:00:44.888994 9200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.60:46775 every 8 connection(s)
I20260620 15:00:44.890475 9047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
I20260620 15:00:44.894012 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 9047
I20260620 15:00:44.894301 9201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:44.904785 9201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.60@KRBTEST.COM
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967644, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.60@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:44.920516 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.896057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.2.70.60:46775)
0620 15:00:44.896794 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.896809 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.896861 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:00:44.899161 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.899189 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.899701 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.900136 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.901438 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.903397 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.905017 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.906126 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.906295 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.911121 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.911130 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.911143 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.911187 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.915609 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.916438 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.916447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.916457 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.916666 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.917061 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.917069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.917072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.917728 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.918023 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.918535 (+ 512us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.919024 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":297,"threads_started":1}
I20260620 15:00:44.923278 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.907887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.2.70.62:40617)
0620 15:00:44.908096 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.908104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.908665 (+ 561us) server_negotiation.cc:407] Connection header received
0620 15:00:44.908844 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.908850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.908939 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.909053 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.910445 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.911245 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.913284 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.913731 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.913829 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.917462 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.917468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.917472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.917536 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.920738 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.921412 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.921418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.921421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.921600 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.921950 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.921956 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.921960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.922263 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.922367 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.922899 (+ 532us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.923060 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:00:44.923633 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.907688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45426)
0620 15:00:44.908522 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.908563 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.908713 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.909118 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.909122 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.909970 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.910304 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.910317 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.911390 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.911394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.913113 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.913129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.913885 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.913890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.914083 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.914755 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.914771 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.917280 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.920871 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.920880 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.920885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.921288 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.921716 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.921720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.921722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.921831 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.922423 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.922432 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.922626 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.922835 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.923139 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":479,"threads_started":1}
I20260620 15:00:44.924888 9201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967644, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.60@KRBTEST.COM for oryx/127.2.70.61@KRBTEST.COM
I20260620 15:00:44.942821 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.925623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.0.0.1:34236)
0620 15:00:44.925824 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.925846 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.925952 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.931689 (+ 5737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.931693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.931994 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.932232 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.932240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.933451 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.933454 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.934399 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.934409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.935132 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.935136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.935266 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.935808 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.935824 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.938170 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.940724 (+ 2554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.940728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.940730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.941081 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.941527 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.941530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.941533 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.941652 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.942101 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.942104 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.942211 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.942310 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.942565 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:00:44.942869 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.925726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34236 (local address 127.2.70.61:36919)
0620 15:00:44.931403 (+ 5677us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.931408 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.931423 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:00:44.931470 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.931473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.931537 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.931626 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.932357 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.933301 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.934524 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.934976 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.935076 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.938350 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.938355 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.938359 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.938388 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.940617 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.941194 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.941197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.941199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.941314 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.941752 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.941755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.941757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.941988 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.942110 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.942409 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.942553 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5567,"thread_start_us":5460,"threads_started":1}
I20260620 15:00:44.943873 9201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:00:44 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967644, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.60@KRBTEST.COM for oryx/127.2.70.60@KRBTEST.COM
I20260620 15:00:44.955655 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.944541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.0.0.1:44262)
0620 15:00:44.944695 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.944709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.944785 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.945088 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.945091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.945424 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:44.945585 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.945602 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.946482 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.946486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.947320 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.947331 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.948029 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.948033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.948186 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.948750 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:44.948773 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:44.950903 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:44.953806 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.953810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.953813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.954141 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.954433 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:44.954436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:44.954438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:44.954567 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:44.954992 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:44.954995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:44.955100 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.955201 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.955375 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 15:00:44.955729 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.944524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.2.70.60:46775)
0620 15:00:44.944680 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.944686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.944747 (+ 61us) server_negotiation.cc:407] Connection header received
0620 15:00:44.944865 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.944869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.944917 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.944986 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:44.945712 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.946364 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.947433 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.947862 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.947956 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.951071 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:44.951075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:44.951079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:44.951105 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:44.953698 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.954231 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.954235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.954237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.954332 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:44.954653 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:44.954657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:44.954659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:44.954897 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:44.955009 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.955252 (+ 243us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.955483 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":51}
I20260620 15:00:44.960237 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Bootstrap starting.
I20260620 15:00:44.963771 9201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:44.964775 9201 log.cc:824] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:44.967360 9201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: No bootstrap required, opened a new log
I20260620 15:00:44.974295 9201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } }
I20260620 15:00:44.974668 9201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:44.974784 9201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0caab6fcb3b8497d918f6eee38ac8305, State: Initialized, Role: FOLLOWER
I20260620 15:00:44.975397 9201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } }
I20260620 15:00:44.976280 9208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } } }
I20260620 15:00:44.976691 9208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:44.977717 9201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:44.986285 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.978389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.0.0.1:34248)
0620 15:00:44.978908 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:44.978940 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:44.979083 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:44.979497 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:44.979501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:44.979528 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:44.979788 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.979802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.981086 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.981094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.983456 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:44.983475 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.985404 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.985422 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:44.985641 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.985697 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 15:00:44.985790 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 15:00:44.985934 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":241,"threads_started":1}
I20260620 15:00:44.986949 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:44.978589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.2.70.61:36919)
0620 15:00:44.978812 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:44.978821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:44.979033 (+ 212us) server_negotiation.cc:407] Connection header received
0620 15:00:44.979239 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:44.979244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:44.979339 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:44.979466 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:44.979999 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.980923 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.983646 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:44.984889 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:44.985040 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:44.985303 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:44.986567 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0620 15:00:44.986702 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 15:00:44.987653 9221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:00:44.987782 9221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:00:44.988130 9095 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260620 15:00:44.988683 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e355101e7d7451493225e2674944836. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } }
I20260620 15:00:44.988903 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: This master's current role is: LEARNER
I20260620 15:00:44.991843 9047 master_runner.cc:428] Detected that this master 0caab6fcb3b8497d918f6eee38ac8305 is joining an existing cluster
I20260620 15:00:44.991976 9047 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.70.60:46775
I20260620 15:00:45.028105 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 8547
I20260620 15:00:45.034467 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.001397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45434 (local address 127.2.70.62:40617)
0620 15:00:45.001585 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.001591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.012287 (+ 10696us) server_negotiation.cc:407] Connection header received
0620 15:00:45.012976 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.012982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.013084 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.013159 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.014720 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.016316 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.018840 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.019470 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.019600 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.022408 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.022414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.022419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.022458 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.026501 (+ 4043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.030607 (+ 4106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.030614 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.030617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.030803 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.031336 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.031342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.031346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.031728 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.031878 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.033806 (+ 1928us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.034056 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:00:45.035562 9229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.000594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45434)
0620 15:00:45.012137 (+ 11543us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.012161 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.012771 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.013394 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.013400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.014146 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:45.014527 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.014543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.016530 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.016536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.018670 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.018685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.019893 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.019900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.020139 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.020917 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:45.020944 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:45.022188 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:45.026675 (+ 4487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.026687 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.026691 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.029866 (+ 3175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.030970 (+ 1104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.030975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.030979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.031160 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.031956 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:45.031961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:45.032188 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.034725 (+ 2537us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.035067 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11354,"mutex_wait_us":32,"thread_start_us":13848,"threads_started":1}
I20260620 15:00:45.036514 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.000577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.2.70.60:46775)
0620 15:00:45.000762 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.000768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.017604 (+ 16836us) server_negotiation.cc:407] Connection header received
0620 15:00:45.017779 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.017783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.017839 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.017906 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.019214 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.020408 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.022371 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.023524 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.023621 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.026025 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.026030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.026035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.026065 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.029664 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.032464 (+ 2800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.032469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.032473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.032619 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.033651 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.033657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.033660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.033980 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.034130 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.035926 (+ 1796us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.036155 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":53}
I20260620 15:00:45.038296 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.000455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.2.70.61:36919)
0620 15:00:45.000652 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.000659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.014417 (+ 13758us) server_negotiation.cc:407] Connection header received
0620 15:00:45.015079 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.015088 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.015195 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.015272 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.017658 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.018764 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.021162 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.021867 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.021979 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.024862 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.024870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.024874 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.024929 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.031668 (+ 6739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.033322 (+ 1654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.033328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.033332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.033503 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.034873 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.034878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.034882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.035235 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.035355 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.037690 (+ 2335us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.037933 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:00:45.039250 9227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.000676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.0.0.1:34256)
0620 15:00:45.014267 (+ 13591us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.014292 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.014869 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.015523 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.015531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.016517 (+ 986us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:45.017455 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.017467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.019269 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.019276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.021010 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.021022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.022252 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.022258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.022432 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.023107 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:45.023131 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:45.024614 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:45.031894 (+ 7280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.031902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.031908 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.032909 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.034211 (+ 1302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.034216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.034220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.034388 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.035447 (+ 1059us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:45.035452 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:45.035657 (+ 205us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.038469 (+ 2812us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.038801 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13370,"mutex_wait_us":314,"thread_start_us":16231,"threads_started":1}
I20260620 15:00:45.037443 9228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.001037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.0.0.1:44270)
0620 15:00:45.017518 (+ 16481us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.017537 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.017674 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.018069 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.018075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.018733 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:45.019079 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.019090 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.020613 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.020618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.022241 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.022251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.023893 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.023899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.024073 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.024685 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:45.024712 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:45.025846 (+ 1134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:45.029979 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.029984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.029988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.030473 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.033059 (+ 2586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.033064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.033067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.033197 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.034513 (+ 1316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:45.034517 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:45.034684 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.036670 (+ 1986us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.036939 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16317,"thread_start_us":11289,"threads_started":1}
Jun 20 15:00:45 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.1@KRBTEST.COM
Jun 20 15:00:45 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967642, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.62@KRBTEST.COM for oryx/127.2.70.60@KRBTEST.COM
I20260620 15:00:45.055853 9230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.029902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.2.70.1:40921)
0620 15:00:45.033213 (+ 3311us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.033220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.033244 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:00:45.033295 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.033302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.033606 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.035876 (+ 2270us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.036133 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.037905 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.041360 (+ 3455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.042375 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.042527 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.047311 (+ 4784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.047325 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.047344 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.047388 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.051493 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.052367 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.052377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.052381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.052533 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.053150 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.053157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.053162 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.053648 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.055021 (+ 1373us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.055180 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.055441 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3121,"thread_start_us":2519,"threads_started":1}
I20260620 15:00:45.056099 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.041687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.2.70.60:46775)
0620 15:00:45.041866 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.041872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.041889 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:00:45.041970 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.041975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.042085 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.042180 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.043232 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.044752 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.046175 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.046665 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.046803 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.050220 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.050224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.050227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.050255 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.053524 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.054197 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.054203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.054206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.054324 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.054678 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.054682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.054686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.055013 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.055152 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.055518 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.055717 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:00:45.056547 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.041374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.0.0.1:44274)
0620 15:00:45.041586 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.041605 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.041732 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.042375 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.042381 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.042884 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:45.043072 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.043083 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.044912 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.044917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.046037 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.046048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.046813 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.046819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.046995 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.047524 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:45.047540 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:45.049977 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:45.053730 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.053734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.053737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.054078 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.054445 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.054451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.054460 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.054568 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.055128 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:45.055133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:45.055293 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.055429 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.055696 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:00:45.057324 8670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:45.058239 8485 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.70.60:46775: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305"
member_type: NON_VOTER
last_known_addr {
host: "127.2.70.60"
port: 46775
}
attrs {
promote: true
}
}
dest_uuid: "3e355101e7d7451493225e2674944836"
cas_config_opid_index: 5
I20260620 15:00:45.059619 8485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } }
I20260620 15:00:45.062779 9232 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0caab6fcb3b8497d918f6eee38ac8305 (127.2.70.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.063850 9176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:45.066695 9232 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.70.60:46775
I20260620 15:00:45.067018 9209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.067402 9209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:45.067438 9095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 LEARNER]: Refusing update from remote peer 3e355101e7d7451493225e2674944836: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 15:00:45.069664 9232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 15:00:45.076607 9095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0caab6fcb3b8497d918f6eee38ac8305 (127.2.70.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.080327 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: NON_VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: true } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.080610 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: This master's current role is: LEARNER
I20260620 15:00:45.081040 9209 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: attempting to promote NON_VOTER 4d3023623215480aa17e5317c6334491 to VOTER
I20260620 15:00:45.081825 9209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } }
I20260620 15:00:45.084470 9095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 LEARNER]: Refusing update from remote peer 3e355101e7d7451493225e2674944836: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 15:00:45.085033 9232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 15:00:45.086083 9176 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Refusing update from remote peer 3e355101e7d7451493225e2674944836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 15:00:45.086656 9047 master.cc:537] Master@127.2.70.60:46775 shutting down...
I20260620 15:00:45.087956 9232 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4d3023623215480aa17e5317c6334491 (127.2.70.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.088521 9095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4d3023623215480aa17e5317c6334491 (127.2.70.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.092058 9175 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Refusing update from remote peer 3e355101e7d7451493225e2674944836: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 15:00:45.092964 9123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.093201 9123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:45.101009 9209 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.004s
I20260620 15:00:45.104612 9047 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:45.105804 9047 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:45.106065 9047 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: stopping tablet replica
I20260620 15:00:45.118201 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.117668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.0.0.1:44276)
0620 15:00:45.117827 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.117940 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 15:00:45.119133 9232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.119444 9047 master.cc:552] Master@127.2.70.60:46775 shutdown complete.
I20260620 15:00:45.119519 9047 master_runner.cc:305] Clearing existing system tablet
I20260620 15:00:45.119957 9047 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:45.120074 9047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:45.120105 9232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:45.119784 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.120579 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:45.125736 9047 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:45.129835 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.129253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.0.0.1:44288)
0620 15:00:45.129504 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.129622 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:00:45.130169 8470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 -> Peer 0caab6fcb3b8497d918f6eee38ac8305 (127.2.70.60:46775): Couldn't send request to peer 0caab6fcb3b8497d918f6eee38ac8305. Status: Network error: Client connection negotiation failed: client connection to 127.2.70.60:46775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:00:45.133857 9239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:45.134241 9047 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:45.134502 9047 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.006s
I20260620 15:00:45.134605 9047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
uuid: "0caab6fcb3b8497d918f6eee38ac8305"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "b4cae699e615e74bb91d1d9b1c2a9745"
server_key_iv: "1d6ca5f22e50db0fe7cf635c813f1a63"
server_key_version: "encryptionkey@0"
I20260620 15:00:45.138213 9047 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:45.142971 9047 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 15:00:45.143131 9047 log.cc:1196] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 15:00:45.143580 9047 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Deleting consensus metadata
I20260620 15:00:45.143824 9047 master_runner.cc:315] Copying system tablet from 127.2.70.62:40617
I20260620 15:00:45.146494 9047 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: tablet copy: Beginning tablet copy session from remote peer at address 127.2.70.62:40617
I20260620 15:00:45.159147 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.147259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45450 (local address 127.2.70.62:40617)
0620 15:00:45.147512 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.147522 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.148013 (+ 491us) server_negotiation.cc:407] Connection header received
0620 15:00:45.148216 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.148221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.148321 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.148438 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.149391 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.150279 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.151475 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.151888 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.152025 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.153901 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.153907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.153946 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.153976 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.156599 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.157222 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.157228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.157232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.157360 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.157767 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.157774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.157778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.158044 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.158202 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.158574 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.158784 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260620 15:00:45.159518 9245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.147251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45450)
0620 15:00:45.147892 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.147922 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.148080 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.148615 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.148618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.149047 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:45.149249 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.149257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.150428 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.150434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.151319 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.151333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.152038 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.152045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.152223 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.152876 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:45.152894 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:45.153707 (+ 813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:45.156734 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.156741 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.156746 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.157084 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.157475 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.157507 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.157514 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.157661 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.158158 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:45.158163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:45.158326 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.158465 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.158765 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":417,"threads_started":1}
I20260620 15:00:45.160097 8506 tablet_copy_service.cc:140] P 3e355101e7d7451493225e2674944836: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0caab6fcb3b8497d918f6eee38ac8305 ({username='oryx', principal='oryx/127.2.70.60@KRBTEST.COM'} at 127.0.0.1:45450)
I20260620 15:00:45.160279 8506 tablet_copy_service.cc:161] P 3e355101e7d7451493225e2674944836: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0caab6fcb3b8497d918f6eee38ac8305 at {username='oryx', principal='oryx/127.2.70.60@KRBTEST.COM'} at 127.0.0.1:45450: session id = 0caab6fcb3b8497d918f6eee38ac8305-00000000000000000000000000000000
I20260620 15:00:45.162979 8506 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 15:00:45.164525 9047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 15:00:45.169039 9047 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: tablet copy: Starting download of 0 data blocks...
I20260620 15:00:45.169312 9047 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: tablet copy: Starting download of 1 WAL segments...
I20260620 15:00:45.171808 9047 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 15:00:45.176385 8506 tablet_copy_service.cc:342] P 3e355101e7d7451493225e2674944836: Request end of tablet copy session 0caab6fcb3b8497d918f6eee38ac8305-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.70.60@KRBTEST.COM'} at 127.0.0.1:45450
I20260620 15:00:45.176540 8506 tablet_copy_service.cc:433] P 3e355101e7d7451493225e2674944836: ending tablet copy session 0caab6fcb3b8497d918f6eee38ac8305-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0caab6fcb3b8497d918f6eee38ac8305
I20260620 15:00:45.189627 9047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:45.192931 9246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:45.193449 9247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:45.200450 9249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:45.201385 9047 server_base.cc:1034] running on GCE node
I20260620 15:00:45.201797 9047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:45.202625 9047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:45.203817 9047 hybrid_clock.cc:648] HybridClock initialized: now 1781967645203782 us; error 24 us; skew 500 ppm
I20260620 15:00:45.204051 9047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:45 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967645, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:45.206677 9047 init.cc:377] Logged in from keytab as oryx/127.2.70.60@KRBTEST.COM (short username oryx)
I20260620 15:00:45.218003 9047 webserver.cc:460] Webserver started at http://127.2.70.60:35517/ using document root <none> and password file <none>
I20260620 15:00:45.218253 9047 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:45.218318 9047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:45.218564 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 8547
I20260620 15:00:45.226178 9047 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.007s
I20260620 15:00:45.228065 9256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:45.228464 9047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 15:00:45.228616 9047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
uuid: "0caab6fcb3b8497d918f6eee38ac8305"
format_stamp: "Formatted at 2026-06-20 15:00:44 on dist-test-slave-n5vv"
server_key: "b4cae699e615e74bb91d1d9b1c2a9745"
server_key_iv: "1d6ca5f22e50db0fe7cf635c813f1a63"
server_key_version: "encryptionkey@0"
I20260620 15:00:45.228806 9047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:45.241290 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:40921
--local_ip_for_outbound_sockets=127.2.70.1
--tserver_master_addrs=127.2.70.62:40617
--webserver_port=37371
--webserver_interface=127.2.70.1
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab
--principal=oryx/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919,127.2.70.60:46775 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:45.246166 9047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:45.250108 9047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:45.277626 9047 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.60:46775
I20260620 15:00:45.277652 9308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.60:46775 every 8 connection(s)
I20260620 15:00:45.278491 9047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
I20260620 15:00:45.283967 9309 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:00:45.286405 9309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Bootstrap starting.
I20260620 15:00:45.305995 9309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:00:45.306449 9309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: Bootstrap complete.
I20260620 15:00:45.307299 9309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } }
I20260620 15:00:45.307543 9309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0caab6fcb3b8497d918f6eee38ac8305, State: Initialized, Role: LEARNER
I20260620 15:00:45.307794 9309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } }
I20260620 15:00:45.309974 9312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.310355 9309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:45.310405 9312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: This master's current role is: LEARNER
I20260620 15:00:45.331255 9323 catalog_manager.cc:1259] Loaded cluster ID: bf8a40f7f97a4ad88e5e36747ef37dc7
I20260620 15:00:45.331377 9323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: loading cluster ID for follower catalog manager: success
I20260620 15:00:45.333842 9323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: acquiring CA information for follower catalog manager: success
I20260620 15:00:45.335723 9323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 15:00:45.636363 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:45.636658 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:45.636857 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:45.645416 9261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:45.645620 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:45.645818 9261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:45.656438 9261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab
--principal=oryx/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:40921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=37371
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919,127.2.70.60:46775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:45.657874 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:45.659196 9261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:45.673630 9331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:45.673720 9328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:45.673697 9329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:45.675098 9261 server_base.cc:1034] running on GCE node
I20260620 15:00:45.675755 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:45.677211 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:45.678484 9261 hybrid_clock.cc:648] HybridClock initialized: now 1781967645678405 us; error 51 us; skew 500 ppm
I20260620 15:00:45.678984 9261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:45 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967645, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:45.683363 9261 init.cc:377] Logged in from keytab as oryx/127.2.70.1@KRBTEST.COM (short username oryx)
I20260620 15:00:45.684991 9261 webserver.cc:460] Webserver started at http://127.2.70.1:37371/ using document root <none> and password file <none>
I20260620 15:00:45.685873 9261 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:45.686035 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:45.688019 9334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.676564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.0.0.1:44296)
0620 15:00:45.677041 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.677064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.677196 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.682207 (+ 5011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.682212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.682245 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:45.682485 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.682502 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.684015 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.684023 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.685416 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.685429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.687222 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.687231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.687420 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.687463 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.687572 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.687714 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260620 15:00:45.688107 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.676702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.2.70.60:46775)
0620 15:00:45.681873 (+ 5171us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.681878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.681895 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:00:45.681959 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.681962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.682031 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.682174 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:45.682678 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.683773 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.685660 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.686919 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.687087 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.687296 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.687619 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.687802 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5049,"thread_start_us":4936,"threads_started":1}
I20260620 15:00:45.689608 9284 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260620 15:00:45.690207 9312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e355101e7d7451493225e2674944836. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:45.690510 9312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: This master's current role is: LEARNER
I20260620 15:00:45.693785 9261 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:45.696550 9341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:45.698200 9261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:45.698357 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "1927c10bb1804fbda5f90a3759744e64"
format_stamp: "Formatted at 2026-06-20 15:00:42 on dist-test-slave-n5vv"
server_key: "52bd33cb843c6281135e5e8b343b5781"
server_key_iv: "59cbc2e9317bdffd85af7229748fe675"
server_key_version: "encryptionkey@0"
I20260620 15:00:45.699431 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:45.720927 9261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:45.725663 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:45.725952 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:45.727056 9261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:45.728684 9261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:45.728794 9261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:45.728938 9261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:45.729034 9261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:45.764528 9261 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:40921
I20260620 15:00:45.764631 9453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:40921 every 8 connection(s)
I20260620 15:00:45.765990 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:45.768981 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 9261
I20260620 15:00:45.769313 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 8681
Jun 20 15:00:45 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.2@KRBTEST.COM
Jun 20 15:00:45 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967645, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
Jun 20 15:00:45 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967645, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.60@KRBTEST.COM
Jun 20 15:00:45 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967645, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.61@KRBTEST.COM
I20260620 15:00:45.805091 9464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.771685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32830 (local address 127.2.70.2:36669)
0620 15:00:45.778024 (+ 6339us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.778037 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.779295 (+ 1258us) server_negotiation.cc:407] Connection header received
0620 15:00:45.779399 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.779411 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.779766 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.779958 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.782844 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.785000 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.787389 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.788312 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.788457 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.794584 (+ 6127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.794599 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.794619 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.794674 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.798729 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.800588 (+ 1859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.800601 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.800606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.800772 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.801219 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.801224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.801229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.803462 (+ 2233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.803757 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.804385 (+ 628us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.804663 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6140,"thread_start_us":6015,"threads_started":1}
I20260620 15:00:45.806797 8804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:45.812245 9463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.771069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:50035 (local address 127.2.70.62:40617)
0620 15:00:45.778409 (+ 7340us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.778416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.786447 (+ 8031us) server_negotiation.cc:407] Connection header received
0620 15:00:45.789781 (+ 3334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.789787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.789873 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.789993 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.793141 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.794302 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.797195 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.797860 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.797974 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.805316 (+ 7342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.805324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.805330 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.805373 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.808940 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.809627 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.809633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.809636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.809766 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.810256 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.810261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.810265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.810536 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.810689 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.811618 (+ 929us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.811850 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7184,"thread_start_us":7100,"threads_started":1}
I20260620 15:00:45.813401 9336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.768954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:55461 (local address 127.2.70.60:46775)
0620 15:00:45.769176 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.769183 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.787023 (+ 17840us) server_negotiation.cc:407] Connection header received
0620 15:00:45.789821 (+ 2798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.789827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.789904 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.789980 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.793141 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.794248 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.797196 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.797868 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.797978 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.803605 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.803610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.803613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.803646 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.808009 (+ 4363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.809191 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.809200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.809206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.809380 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.810139 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.810147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.810152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.811163 (+ 1011us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.811331 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.812073 (+ 742us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.812917 (+ 844us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 15:00:45.813505 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.770093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:52807 (local address 127.2.70.61:36919)
0620 15:00:45.785559 (+ 15466us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:45.785564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:45.787749 (+ 2185us) server_negotiation.cc:407] Connection header received
0620 15:00:45.789820 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:45.789827 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:45.789925 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:45.790008 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:45.793146 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.794320 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.797302 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.798021 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.798122 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.804316 (+ 6194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:45.804324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:45.804331 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:45.804375 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:45.807820 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.808896 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.808902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.808905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.809055 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:45.810818 (+ 1763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:45.810823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:45.810826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:45.811214 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:45.811332 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:45.812900 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0620 15:00:45.813149 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15319,"thread_start_us":15217,"threads_started":1}
I20260620 15:00:45.813557 9462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.770016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.1:50035)
0620 15:00:45.786271 (+ 16255us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.786336 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.789399 (+ 3063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.790979 (+ 1580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.790984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.791825 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:45.792937 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.792962 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.794834 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.794839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.797033 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.797047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.798511 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.798517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.798682 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.799908 (+ 1226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:45.799957 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:45.805064 (+ 5107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:45.809177 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.809180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.809182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.809476 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.809894 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.809898 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.809902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.810152 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.810673 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:45.810681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:45.811209 (+ 528us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.811522 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.812044 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15905,"mutex_wait_us":432,"spinlock_wait_cycles":27520,"thread_start_us":17233,"threads_started":1}
I20260620 15:00:45.815055 9455 heartbeater.cc:347] Connected to a master server at 127.2.70.62:40617
I20260620 15:00:45.815232 9459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.770659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.2.70.1:52807)
0620 15:00:45.787660 (+ 17001us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.787677 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.789418 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.790558 (+ 1140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.790571 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.791943 (+ 1372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:45.792937 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.792963 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.794550 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.794558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.797132 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.797149 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.798482 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.798504 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.798749 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.800083 (+ 1334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:45.800105 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:45.804053 (+ 3948us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:45.807989 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.807998 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.808014 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.808757 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.809952 (+ 1195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.809956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.809959 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.810189 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.811418 (+ 1229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:45.811422 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:45.812607 (+ 1185us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.813740 (+ 1133us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.814472 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16872,"mutex_wait_us":645,"spinlock_wait_cycles":28288,"thread_start_us":15147,"threads_started":1}
I20260620 15:00:45.815373 9455 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:45.815827 9460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:45.770016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.2.70.1:55461)
0620 15:00:45.786920 (+ 16904us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:45.786939 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:45.789399 (+ 2460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:45.790557 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:45.791007 (+ 450us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:45.791889 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:45.792937 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.792962 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.794544 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.794549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.797034 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:45.797047 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:45.798299 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:45.798308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:45.798660 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:45.799896 (+ 1236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:45.799927 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:45.803373 (+ 3446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:45.808265 (+ 4892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.808273 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.808280 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.809028 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.809581 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:45.809589 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:45.809595 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:45.809978 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:45.811468 (+ 1490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:45.811476 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:45.811704 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 15:00:45.814412 (+ 2708us) client_negotiation.cc:240] Negotiation successful
0620 15:00:45.814807 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16735,"mutex_wait_us":951,"spinlock_wait_cycles":22784,"thread_start_us":16629,"threads_started":1}
I20260620 15:00:45.816192 9455 heartbeater.cc:511] Master 127.2.70.62:40617 requested a full tablet report, sending...
I20260620 15:00:45.817338 9454 heartbeater.cc:347] Connected to a master server at 127.2.70.60:46775
I20260620 15:00:45.817538 9454 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:45.817945 8485 ts_manager.cc:194] Re-registered known tserver with Master: 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921)
I20260620 15:00:45.818063 9454 heartbeater.cc:511] Master 127.2.70.60:46775 requested a full tablet report, sending...
I20260620 15:00:45.818693 8485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:50035
I20260620 15:00:45.819221 9456 heartbeater.cc:347] Connected to a master server at 127.2.70.61:36919
I20260620 15:00:45.819337 9456 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:45.819698 9456 heartbeater.cc:511] Master 127.2.70.61:36919 requested a full tablet report, sending...
I20260620 15:00:45.820451 9274 ts_manager.cc:194] Registered new tserver with Master: 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921)
I20260620 15:00:45.821255 9085 ts_manager.cc:194] Registered new tserver with Master: 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921)
I20260620 15:00:45.905301 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 8681
I20260620 15:00:45.921530 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:36669
--local_ip_for_outbound_sockets=127.2.70.2
--tserver_master_addrs=127.2.70.62:40617
--webserver_port=45885
--webserver_interface=127.2.70.2
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab
--principal=oryx/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919,127.2.70.60:46775 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:46.184288 9332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: NON_VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: true } } }
I20260620 15:00:46.184749 9332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:46.186357 9332 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: attempting to promote NON_VOTER 0caab6fcb3b8497d918f6eee38ac8305 to VOTER
I20260620 15:00:46.186895 9332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } }
I20260620 15:00:46.189216 9284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 LEARNER]: Refusing update from remote peer 3e355101e7d7451493225e2674944836: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 15:00:46.189893 9467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 15:00:46.190363 9095 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Refusing update from remote peer 3e355101e7d7451493225e2674944836: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260620 15:00:46.191006 9467 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 15:00:46.193861 9468 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0caab6fcb3b8497d918f6eee38ac8305 (127.2.70.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:46.194545 9095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0caab6fcb3b8497d918f6eee38ac8305 (127.2.70.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:46.194576 9284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 0caab6fcb3b8497d918f6eee38ac8305 (127.2.70.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:46.198338 9468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:46.198709 9468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:46.198647 9467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:46.199048 9467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:46.199044 9468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:46.199411 9468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:46.205706 9474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:46.206136 9474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:46.207058 9473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3e355101e7d7451493225e2674944836" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:46.207451 9473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 15:00:46.260993 9466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:46.261282 9466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:46.261394 9466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:46.268930 9466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:46.269040 9466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:46.269152 9466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:46.279246 9466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab
--principal=oryx/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:36669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=45885
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919,127.2.70.60:46775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:46.280611 9466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:46.282090 9466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:46.295233 9479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:46.295257 9480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:46.295192 9482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:46.295917 9466 server_base.cc:1034] running on GCE node
I20260620 15:00:46.296531 9466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:46.297719 9466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:46.298923 9466 hybrid_clock.cc:648] HybridClock initialized: now 1781967646298858 us; error 63 us; skew 500 ppm
I20260620 15:00:46.299280 9466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:46 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967646, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:46.302696 9466 init.cc:377] Logged in from keytab as oryx/127.2.70.2@KRBTEST.COM (short username oryx)
I20260620 15:00:46.304261 9466 webserver.cc:460] Webserver started at http://127.2.70.2:45885/ using document root <none> and password file <none>
I20260620 15:00:46.304965 9466 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:46.305099 9466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:46.310626 9466 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:00:46.313550 9489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:46.315115 9466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:00:46.315284 9466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "7c6f288b292f48d9a59f76e4d97503c0"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-n5vv"
server_key: "228acc7d36a092fcb4da62078bd1745c"
server_key_iv: "83416259afcc94b231ad76526037fe97"
server_key_version: "encryptionkey@0"
I20260620 15:00:46.316287 9466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:46.338662 9466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:46.342810 9466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:46.343091 9466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:46.344100 9466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:46.345677 9466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:46.345809 9466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:46.345927 9466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:46.346007 9466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:46.380183 9466 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:36669
I20260620 15:00:46.380282 9601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:36669 every 8 connection(s)
I20260620 15:00:46.381852 9466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:46.382719 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 9466
I20260620 15:00:46.383056 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 8814
Jun 20 15:00:46 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967646, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.61@KRBTEST.COM
Jun 20 15:00:46 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967646, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.60@KRBTEST.COM
Jun 20 15:00:46 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.3@KRBTEST.COM
Jun 20 15:00:46 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967646, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:46.426748 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:46.387103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:33543 (local address 127.2.70.60:46775)
0620 15:00:46.397869 (+ 10766us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:46.397878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:46.399021 (+ 1143us) server_negotiation.cc:407] Connection header received
0620 15:00:46.403348 (+ 4327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:46.403355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:46.403449 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:46.403559 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:46.406582 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.407969 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.411607 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.412370 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.412514 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:46.418656 (+ 6142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:46.418664 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:46.418671 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:46.418714 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:46.422773 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:46.423742 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:46.423750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:46.423754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:46.423914 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:46.424388 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:46.424393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:46.424397 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:46.424806 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:46.424983 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:46.426029 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0620 15:00:46.426264 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10613,"thread_start_us":10518,"threads_started":1}
I20260620 15:00:46.428346 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:46.385796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.2.70.2:55703)
0620 15:00:46.403602 (+ 17806us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:46.403621 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:46.403726 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:46.407483 (+ 3757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:46.407487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:46.408020 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:46.409116 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:46.409130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.410359 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.410365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:46.412050 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:46.412064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.413224 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.413234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:46.413471 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:46.414215 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:46.414243 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:46.417767 (+ 3524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:46.421970 (+ 4203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:46.421980 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:46.422004 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:46.422705 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:46.423152 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:46.423158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:46.423162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:46.423424 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:46.424158 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:46.424166 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:46.424602 (+ 436us) client_negotiation.cc:769] Sending connection context
0620 15:00:46.424942 (+ 340us) client_negotiation.cc:240] Negotiation successful
0620 15:00:46.425522 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17642,"mutex_wait_us":2166,"thread_start_us":10062,"threads_started":1}
I20260620 15:00:46.428380 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:46.386612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.2.70.2:33543)
0620 15:00:46.398390 (+ 11778us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:46.398415 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:46.402786 (+ 4371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:46.404085 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:46.404099 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:46.405204 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:46.406379 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:46.406400 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.408180 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.408186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:46.411412 (+ 3226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:46.411431 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.412820 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.412829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:46.413121 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:46.414476 (+ 1355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:46.414508 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:46.418395 (+ 3887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:46.422942 (+ 4547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:46.422948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:46.422952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:46.423579 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:46.424069 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:46.424074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:46.424080 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:46.424252 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:46.425052 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:46.425058 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:46.425242 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 15:00:46.425430 (+ 188us) client_negotiation.cc:240] Negotiation successful
0620 15:00:46.425743 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11621,"thread_start_us":7865,"threads_started":1}
I20260620 15:00:46.428979 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:46.386093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:55703 (local address 127.2.70.61:36919)
0620 15:00:46.407212 (+ 21119us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:46.407217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:46.407231 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:00:46.407276 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:46.407279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:46.407333 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:46.407458 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:46.409303 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.410221 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.412222 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.412851 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.412957 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:46.418036 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:46.418044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:46.418049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:46.418102 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:46.421807 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:46.422851 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:46.422858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:46.422863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:46.422997 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:46.423583 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:46.423587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:46.423590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:46.423952 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:46.424077 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:46.426107 (+ 2030us) server_negotiation.cc:299] Negotiation successful
0620 15:00:46.426743 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21026,"thread_start_us":21479,"threads_started":1}
I20260620 15:00:46.430097 9604 heartbeater.cc:347] Connected to a master server at 127.2.70.61:36919
I20260620 15:00:46.430227 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:46.384840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.2.70.3:39307)
0620 15:00:46.404700 (+ 19860us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:46.404711 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:46.405474 (+ 763us) server_negotiation.cc:407] Connection header received
0620 15:00:46.405580 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:46.405589 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:46.405843 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:46.406012 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:46.408158 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.409717 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.411815 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.412694 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.412819 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:46.420306 (+ 7487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:46.420328 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:46.420349 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:46.420400 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:46.424969 (+ 4569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:46.425836 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:46.425846 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:46.425849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:46.426023 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:46.427262 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:46.427267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:46.427271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:46.427695 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:46.428023 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:46.429413 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0620 15:00:46.429717 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19666,"thread_start_us":19869,"threads_started":1}
I20260620 15:00:46.430426 9604 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:46.430749 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:46.385307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:49637 (local address 127.2.70.62:40617)
0620 15:00:46.404772 (+ 19465us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:46.404778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:46.404798 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:00:46.407900 (+ 3102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:46.407906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:46.408010 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:46.408115 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:46.410418 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.411579 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.413789 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.414378 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.414534 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:46.422560 (+ 8026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:46.422568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:46.422575 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:46.422632 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:46.426707 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:46.427696 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:46.427702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:46.427706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:46.427866 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:46.428430 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:46.428438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:46.428441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:46.428876 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:46.429049 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:46.430092 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0620 15:00:46.430324 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18858,"thread_start_us":18751,"threads_started":1}
I20260620 15:00:46.431281 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:46.385796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.2:49637)
0620 15:00:46.394916 (+ 9120us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:46.394979 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:46.407694 (+ 12715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:46.408703 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:46.408711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:46.409317 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:46.409684 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:46.409698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.411850 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.411856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:46.413631 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:46.413645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:46.414859 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:46.414867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:46.415121 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:46.415798 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:46.415830 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:46.422234 (+ 6404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:46.426883 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:46.426887 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:46.426892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:46.427550 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:46.428025 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:46.428031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:46.428035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:46.428252 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:46.429109 (+ 857us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:46.429113 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:46.429268 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 15:00:46.429405 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 15:00:46.429737 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8801,"mutex_wait_us":40,"thread_start_us":17285,"threads_started":1}
I20260620 15:00:46.431370 9604 heartbeater.cc:511] Master 127.2.70.61:36919 requested a full tablet report, sending...
I20260620 15:00:46.432191 8937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:46.432204 9603 heartbeater.cc:347] Connected to a master server at 127.2.70.62:40617
I20260620 15:00:46.432366 9603 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:46.432818 9603 heartbeater.cc:511] Master 127.2.70.62:40617 requested a full tablet report, sending...
I20260620 15:00:46.432229 9602 heartbeater.cc:347] Connected to a master server at 127.2.70.60:46775
I20260620 15:00:46.433197 9602 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:46.433645 9602 heartbeater.cc:511] Master 127.2.70.60:46775 requested a full tablet report, sending...
I20260620 15:00:46.433682 9085 ts_manager.cc:194] Registered new tserver with Master: 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2:36669)
I20260620 15:00:46.434139 8485 ts_manager.cc:194] Re-registered known tserver with Master: 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2:36669)
I20260620 15:00:46.434738 9274 ts_manager.cc:194] Registered new tserver with Master: 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2:36669)
I20260620 15:00:46.435354 8485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:49637
I20260620 15:00:46.521216 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 8814
I20260620 15:00:46.541222 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:39307
--local_ip_for_outbound_sockets=127.2.70.3
--tserver_master_addrs=127.2.70.62:40617
--webserver_port=36285
--webserver_interface=127.2.70.3
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab
--principal=oryx/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919,127.2.70.60:46775 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:00:46.822204 9455 heartbeater.cc:503] Master 127.2.70.62:40617 was elected leader, sending a full tablet report...
W20260620 15:00:46.880527 9615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:46.880862 9615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:46.881033 9615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:46.888942 9615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:46.889104 9615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:46.889287 9615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:46.900485 9615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab
--principal=oryx/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:39307
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=36285
--tserver_master_addrs=127.2.70.62:40617,127.2.70.61:36919,127.2.70.60:46775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:46.901970 9615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:46.903364 9615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:46.916939 9620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:46.916997 9621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:46.917035 9623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:46.917742 9615 server_base.cc:1034] running on GCE node
I20260620 15:00:46.918378 9615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:46.919466 9615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:46.920718 9615 hybrid_clock.cc:648] HybridClock initialized: now 1781967646920642 us; error 55 us; skew 500 ppm
I20260620 15:00:46.921130 9615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:46 dist-test-slave-n5vv krb5kdc[8441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967646, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:46.924525 9615 init.cc:377] Logged in from keytab as oryx/127.2.70.3@KRBTEST.COM (short username oryx)
I20260620 15:00:46.925973 9615 webserver.cc:460] Webserver started at http://127.2.70.3:36285/ using document root <none> and password file <none>
I20260620 15:00:46.926661 9615 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:46.926812 9615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:46.932711 9615 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:00:46.935496 9630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:46.936856 9615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:00:46.937033 9615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "e48b32dc05114c65889487a5089f96d3"
format_stamp: "Formatted at 2026-06-20 15:00:43 on dist-test-slave-n5vv"
server_key: "809a7bbacdd033e78fab13f236aec82f"
server_key_iv: "9e799a7cb00fb4f4af32138842640593"
server_key_version: "encryptionkey@0"
I20260620 15:00:46.938115 9615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:46.961658 9615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:46.965893 9615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:46.966180 9615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:46.967190 9615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:46.968506 9615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:46.968626 9615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:46.968770 9615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:46.968856 9615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:47.003317 9615 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:39307
I20260620 15:00:47.003340 9742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:39307 every 8 connection(s)
I20260620 15:00:47.004914 9615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:47.012905 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 9615
Jun 20 15:00:47 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967646, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.61@KRBTEST.COM
Jun 20 15:00:47 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967646, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
Jun 20 15:00:47 dist-test-slave-n5vv krb5kdc[8441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967646, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.60@KRBTEST.COM
I20260620 15:00:47.061473 9748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.007915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:51319 (local address 127.2.70.61:36919)
0620 15:00:47.018778 (+ 10863us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.018786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.018808 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:00:47.021046 (+ 2238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.021051 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.021140 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.021248 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.024022 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.025121 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.030042 (+ 4921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.030970 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.031145 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.047485 (+ 16340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.047495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.047501 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.047543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.052186 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.055318 (+ 3132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.055324 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.055327 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.055491 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.056026 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.056033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.056036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.056348 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.056479 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.060773 (+ 4294us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.061028 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10746,"thread_start_us":10900,"threads_started":1}
I20260620 15:00:47.064225 9750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.009422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:52765 (local address 127.2.70.62:40617)
0620 15:00:47.021874 (+ 12452us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.021880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.021899 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:00:47.021959 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.021965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.022053 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.023435 (+ 1382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.024514 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.035454 (+ 10940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.038400 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.039162 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.039340 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.048446 (+ 9106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.048456 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.048463 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.048513 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.054246 (+ 5733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.060930 (+ 6684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.060935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.060939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.061085 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.061554 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.061560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.061563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.061902 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.062024 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.063597 (+ 1573us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.063830 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12336,"thread_start_us":12238,"threads_started":1}
I20260620 15:00:47.068523 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.009859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.2.70.3:51319)
0620 15:00:47.018507 (+ 8648us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:47.018531 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:47.020744 (+ 2213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:47.021763 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:47.021776 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:47.022821 (+ 1045us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:47.023796 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.023823 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.027265 (+ 3442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.027273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.029851 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.029870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.038054 (+ 8184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.038062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.038385 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.040047 (+ 1662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:47.040091 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:47.045559 (+ 5468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:47.054502 (+ 8943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:47.054507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:47.054511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:47.055168 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:47.055624 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:47.055629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:47.055633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:47.055879 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:47.057956 (+ 2077us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:47.057966 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:47.058915 (+ 949us) client_negotiation.cc:769] Sending connection context
0620 15:00:47.060056 (+ 1141us) client_negotiation.cc:240] Negotiation successful
0620 15:00:47.063494 (+ 3438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8486,"mutex_wait_us":30,"spinlock_wait_cycles":17152,"thread_start_us":6404,"threads_started":1}
I20260620 15:00:47.069711 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.008709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.3:52765)
0620 15:00:47.016823 (+ 8114us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:47.016898 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:47.021194 (+ 4296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:47.022228 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:47.022239 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:47.022941 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:47.023796 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.023823 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.036167 (+ 12344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.036176 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.038180 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.038194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.040080 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.040085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.040279 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.042256 (+ 1977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:47.042282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:47.046797 (+ 4515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:47.054438 (+ 7641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:47.054449 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:47.054470 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:47.060769 (+ 6299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:47.061223 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:47.061227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:47.061230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:47.061418 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:47.062445 (+ 1027us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:47.062451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:47.062607 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 15:00:47.062746 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:00:47.069237 (+ 6491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7675,"mutex_wait_us":6029,"thread_start_us":15186,"threads_started":1}
I20260620 15:00:47.069944 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.026190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34258 (local address 127.2.70.61:36919)
0620 15:00:47.047162 (+ 20972us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.047171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.050603 (+ 3432us) server_negotiation.cc:407] Connection header received
0620 15:00:47.050853 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.050857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.050938 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.051019 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.053937 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.055166 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.057014 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.057712 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.057834 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.062355 (+ 4521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.062365 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.062369 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.062420 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.066422 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.067252 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.067257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.067261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.067439 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.067896 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.067901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.067905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.068230 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.068474 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.068870 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.069507 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20806,"thread_start_us":20979,"threads_started":1}
I20260620 15:00:47.071038 9744 heartbeater.cc:347] Connected to a master server at 127.2.70.62:40617
I20260620 15:00:47.071376 9744 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:47.072088 9745 heartbeater.cc:347] Connected to a master server at 127.2.70.61:36919
I20260620 15:00:47.072227 9745 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:47.072430 9744 heartbeater.cc:511] Master 127.2.70.62:40617 requested a full tablet report, sending...
I20260620 15:00:47.072765 9745 heartbeater.cc:511] Master 127.2.70.61:36919 requested a full tablet report, sending...
I20260620 15:00:47.073845 9759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.024971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44310 (local address 127.2.70.60:46775)
0620 15:00:47.047771 (+ 22800us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.047779 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.047801 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:00:47.047865 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.047873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.047978 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.048105 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.052003 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.053292 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.055370 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.055995 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.056115 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.061451 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.061460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.061466 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.061535 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.067019 (+ 5484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.070367 (+ 3348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.070373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.070376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.070551 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.071526 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.071534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.071537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.071867 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.072131 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.073194 (+ 1063us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.073423 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22648,"thread_start_us":22527,"threads_started":1}
I20260620 15:00:47.074306 8485 ts_manager.cc:194] Re-registered known tserver with Master: e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307)
I20260620 15:00:47.074352 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.010375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.2.70.3:49339)
0620 15:00:47.024313 (+ 13938us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:47.024337 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:47.031245 (+ 6908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:47.032418 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:47.032426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:47.034237 (+ 1811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:47.034628 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.034643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.044020 (+ 9377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.044025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.045590 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.045599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.056140 (+ 10541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.056147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.056335 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.057108 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:47.057140 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:47.062280 (+ 5140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:47.067299 (+ 5019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:47.067304 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:47.067308 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:47.069041 (+ 1733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:47.069567 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:47.069572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:47.069576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:47.071208 (+ 1632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:47.071962 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:47.071967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:47.073523 (+ 1556us) client_negotiation.cc:769] Sending connection context
0620 15:00:47.073677 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 15:00:47.073961 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13823,"mutex_wait_us":3,"thread_start_us":7764,"threads_started":1}
I20260620 15:00:47.074326 9085 ts_manager.cc:194] Registered new tserver with Master: e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307)
I20260620 15:00:47.075273 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.007869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:49339 (local address 127.2.70.60:46775)
0620 15:00:47.019181 (+ 11312us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.019189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.024411 (+ 5222us) server_negotiation.cc:407] Connection header received
0620 15:00:47.031879 (+ 7468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.031889 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.032011 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.032112 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.040756 (+ 8644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.041998 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.049449 (+ 7451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.050211 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.050351 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.062494 (+ 12143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.062502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.062509 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.062559 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.067134 (+ 4575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.069192 (+ 2058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.069197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.069200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.069387 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.071352 (+ 1965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.071358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.071363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.071756 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.071891 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.074661 (+ 2770us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.074881 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11195,"thread_start_us":13210,"threads_started":1}
I20260620 15:00:47.075546 8485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:52765
I20260620 15:00:47.076027 9743 heartbeater.cc:347] Connected to a master server at 127.2.70.60:46775
I20260620 15:00:47.076159 9743 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:47.077072 9743 heartbeater.cc:511] Master 127.2.70.60:46775 requested a full tablet report, sending...
I20260620 15:00:47.078260 9274 ts_manager.cc:194] Registered new tserver with Master: e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307)
I20260620 15:00:47.079744 9758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.022799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45466 (local address 127.2.70.62:40617)
0620 15:00:47.050300 (+ 27501us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.050304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.050323 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:00:47.050364 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.050367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.050479 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.050591 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.054738 (+ 4147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.055919 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.059245 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.060060 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.060190 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.066977 (+ 6787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.066982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.066986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.067022 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.076459 (+ 9437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.077762 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.077768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.077771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.077927 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.078358 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.078363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.078367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.078698 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.078894 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.079174 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.079362 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27394,"thread_start_us":27280,"threads_started":1}
I20260620 15:00:47.083365 8485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:47.090415 8485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:47.142632 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.123015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.2.70.3:39307)
0620 15:00:47.134056 (+ 11041us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.134068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.134667 (+ 599us) server_negotiation.cc:407] Connection header received
0620 15:00:47.134851 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.134861 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.135208 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.135429 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:47.136004 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.137056 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.139658 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.141162 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.141291 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.141574 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.142110 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.142264 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10832,"thread_start_us":10654,"threads_started":1}
I20260620 15:00:47.142685 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.122611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:39307 (local address 127.0.0.1:52856)
0620 15:00:47.134534 (+ 11923us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:47.134551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:47.134703 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:47.135408 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:47.135412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:47.135437 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:47.135808 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.135819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.137249 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.137259 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.139467 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.139482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.141725 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.141731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.141941 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.141975 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:00:47.142070 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:00:47.142194 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11806,"mutex_wait_us":49,"thread_start_us":10871,"threads_started":1}
I20260620 15:00:47.144290 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.122380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32846 (local address 127.2.70.2:36669)
0620 15:00:47.133536 (+ 11156us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.133547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.133581 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:00:47.133698 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.135116 (+ 1418us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.135412 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.135609 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:47.137225 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.138646 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.141774 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.143232 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.143355 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.143562 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.143846 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.144013 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10930,"thread_start_us":10835,"threads_started":1}
I20260620 15:00:47.144302 9773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.122111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:36669 (local address 127.0.0.1:32846)
0620 15:00:47.133167 (+ 11056us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:47.133192 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:47.133363 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:47.136651 (+ 3288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:47.136658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:47.136691 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:47.137058 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.137074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.138797 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.138802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.141609 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.141624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.143449 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.143456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.143656 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.143687 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:00:47.143776 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 15:00:47.143892 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10875,"mutex_wait_us":27,"thread_start_us":12143,"threads_started":1}
I20260620 15:00:47.147238 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.135872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.2.70.1:40921)
0620 15:00:47.136371 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.136379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.137164 (+ 785us) server_negotiation.cc:407] Connection header received
0620 15:00:47.138799 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.138808 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.139068 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.139240 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:47.140173 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.142077 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.144722 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.146168 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.146295 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.146468 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.146811 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.146963 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":254,"threads_started":1}
I20260620 15:00:47.147274 9776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.136437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40921 (local address 127.0.0.1:57958)
0620 15:00:47.137027 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:47.137050 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:47.138587 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:47.139494 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:47.139500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:47.139532 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:47.139977 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.139988 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.142264 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.142269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.144564 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.144578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.146437 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.146443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.146618 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.146650 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:00:47.146750 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:00:47.146960 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":753,"threads_started":1}
I20260620 15:00:47.147948 9678 tablet_service.cc:1467] Processing CreateTablet for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d (DEFAULT_TABLE table=test-table [id=7e9d8170ef5a4f8dad75596998c27100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:47.147994 9537 tablet_service.cc:1467] Processing CreateTablet for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d (DEFAULT_TABLE table=test-table [id=7e9d8170ef5a4f8dad75596998c27100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:47.149402 9678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:47.149408 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:47.150847 9389 tablet_service.cc:1467] Processing CreateTablet for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d (DEFAULT_TABLE table=test-table [id=7e9d8170ef5a4f8dad75596998c27100]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:47.152144 9389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:47.161825 9777 tablet_bootstrap.cc:492] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: Bootstrap starting.
I20260620 15:00:47.164943 9778 tablet_bootstrap.cc:492] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Bootstrap starting.
I20260620 15:00:47.166021 9777 tablet_bootstrap.cc:654] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:47.167387 9777 log.cc:824] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:47.168517 9778 tablet_bootstrap.cc:654] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:47.170082 9778 log.cc:824] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:47.170292 9779 tablet_bootstrap.cc:492] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: Bootstrap starting.
I20260620 15:00:47.173339 9779 tablet_bootstrap.cc:654] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:47.174540 9779 log.cc:824] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:47.176182 9778 tablet_bootstrap.cc:492] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: No bootstrap required, opened a new log
I20260620 15:00:47.176185 9777 tablet_bootstrap.cc:492] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: No bootstrap required, opened a new log
I20260620 15:00:47.176446 9778 ts_tablet_manager.cc:1397] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260620 15:00:47.176457 9777 ts_tablet_manager.cc:1397] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260620 15:00:47.177533 9779 tablet_bootstrap.cc:492] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: No bootstrap required, opened a new log
I20260620 15:00:47.177788 9779 ts_tablet_manager.cc:1397] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260620 15:00:47.184036 9777 raft_consensus.cc:348] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.184419 9777 raft_consensus.cc:374] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:47.184511 9777 raft_consensus.cc:729] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6f288b292f48d9a59f76e4d97503c0, State: Initialized, Role: FOLLOWER
I20260620 15:00:47.185127 9777 consensus_queue.cc:260] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.187366 9777 ts_tablet_manager.cc:1428] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260620 15:00:47.187136 9778 raft_consensus.cc:348] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.187559 9778 raft_consensus.cc:374] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:47.187637 9603 heartbeater.cc:503] Master 127.2.70.62:40617 was elected leader, sending a full tablet report...
I20260620 15:00:47.187716 9778 raft_consensus.cc:729] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48b32dc05114c65889487a5089f96d3, State: Initialized, Role: FOLLOWER
I20260620 15:00:47.188328 9778 consensus_queue.cc:260] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.188488 9779 raft_consensus.cc:348] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.189035 9779 raft_consensus.cc:374] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:47.189139 9779 raft_consensus.cc:729] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1927c10bb1804fbda5f90a3759744e64, State: Initialized, Role: FOLLOWER
I20260620 15:00:47.189762 9779 consensus_queue.cc:260] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.190310 9778 ts_tablet_manager.cc:1428] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260620 15:00:47.192221 9779 ts_tablet_manager.cc:1428] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260620 15:00:47.193176 9744 heartbeater.cc:503] Master 127.2.70.62:40617 was elected leader, sending a full tablet report...
I20260620 15:00:47.207682 9785 raft_consensus.cc:482] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:47.208148 9785 raft_consensus.cc:504] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.209769 9785 leader_election.cc:288] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2:36669), e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307)
I20260620 15:00:47.218354 9772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.210172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:37703 (local address 127.2.70.2:36669)
0620 15:00:47.210442 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.210452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.211545 (+ 1093us) server_negotiation.cc:407] Connection header received
0620 15:00:47.211690 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.211694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.211771 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.211838 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:47.212310 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.213293 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.215835 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.217101 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.217242 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.217346 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.218074 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.218167 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 15:00:47.218582 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.209981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:36669 (local address 127.2.70.1:37703)
0620 15:00:47.211378 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:47.211394 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:47.211551 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:47.211871 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:47.211874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:47.211893 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:47.212147 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.212159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.213413 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.213416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.215673 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.215688 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.217410 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.217414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.217773 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.217923 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 15:00:47.218017 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:00:47.218165 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1281,"mutex_wait_us":255,"spinlock_wait_cycles":9216,"thread_start_us":995,"threads_started":1}
I20260620 15:00:47.218914 9786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.209981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:39307 (local address 127.2.70.1:50317)
0620 15:00:47.211359 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:47.211394 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:47.211551 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:47.212176 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:47.212185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:47.212208 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:47.212479 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.212487 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.213827 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.213832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.215846 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:47.215858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.217396 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.217403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:47.217771 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.217923 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 15:00:47.218427 (+ 504us) client_negotiation.cc:240] Negotiation successful
0620 15:00:47.218565 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1232,"mutex_wait_us":167,"spinlock_wait_cycles":3712,"thread_start_us":940,"threads_started":1}
I20260620 15:00:47.218997 9774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.210219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:50317 (local address 127.2.70.3:39307)
0620 15:00:47.210820 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.210828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.211498 (+ 670us) server_negotiation.cc:407] Connection header received
0620 15:00:47.212015 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.212018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.212072 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.212137 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:47.212608 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.213708 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.215979 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.217160 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.217268 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.217355 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.218664 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.218810 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 15:00:47.219849 9698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6d77dc2cdb4844ab2bd1ec371aab7d" candidate_uuid: "1927c10bb1804fbda5f90a3759744e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48b32dc05114c65889487a5089f96d3" is_pre_election: true
I20260620 15:00:47.219981 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6d77dc2cdb4844ab2bd1ec371aab7d" candidate_uuid: "1927c10bb1804fbda5f90a3759744e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6f288b292f48d9a59f76e4d97503c0" is_pre_election: true
I20260620 15:00:47.220371 9698 raft_consensus.cc:2454] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1927c10bb1804fbda5f90a3759744e64 in term 0.
I20260620 15:00:47.220523 9557 raft_consensus.cc:2454] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1927c10bb1804fbda5f90a3759744e64 in term 0.
I20260620 15:00:47.221122 9344 leader_election.cc:302] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1927c10bb1804fbda5f90a3759744e64, e48b32dc05114c65889487a5089f96d3; no voters:
I20260620 15:00:47.221580 9785 raft_consensus.cc:2781] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:47.221743 9785 raft_consensus.cc:482] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:47.221863 9785 raft_consensus.cc:3037] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:47.224417 9785 raft_consensus.cc:504] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.224937 9785 leader_election.cc:288] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [CANDIDATE]: Term 1 election: Requested vote from peers 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2:36669), e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307)
I20260620 15:00:47.225337 9557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6d77dc2cdb4844ab2bd1ec371aab7d" candidate_uuid: "1927c10bb1804fbda5f90a3759744e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6f288b292f48d9a59f76e4d97503c0"
I20260620 15:00:47.225373 9698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6d77dc2cdb4844ab2bd1ec371aab7d" candidate_uuid: "1927c10bb1804fbda5f90a3759744e64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48b32dc05114c65889487a5089f96d3"
I20260620 15:00:47.225546 9557 raft_consensus.cc:3037] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:47.225595 9698 raft_consensus.cc:3037] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:47.227735 9698 raft_consensus.cc:2454] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1927c10bb1804fbda5f90a3759744e64 in term 1.
I20260620 15:00:47.227737 9557 raft_consensus.cc:2454] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1927c10bb1804fbda5f90a3759744e64 in term 1.
I20260620 15:00:47.228159 9344 leader_election.cc:302] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1927c10bb1804fbda5f90a3759744e64, 7c6f288b292f48d9a59f76e4d97503c0, e48b32dc05114c65889487a5089f96d3; no voters:
I20260620 15:00:47.228506 9785 raft_consensus.cc:2781] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:47.228987 9785 raft_consensus.cc:686] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 1 LEADER]: Becoming Leader. State: Replica: 1927c10bb1804fbda5f90a3759744e64, State: Running, Role: LEADER
I20260620 15:00:47.229607 9785 consensus_queue.cc:237] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } }
I20260620 15:00:47.234905 8485 catalog_manager.cc:5495] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "1927c10bb1804fbda5f90a3759744e64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:47.255338 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.250132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.2.70.1:40921)
0620 15:00:47.250328 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.250334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.250352 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:00:47.250553 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.250565 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.250650 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.250796 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:47.251157 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.251865 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.253194 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.253675 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.253810 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.254272 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:47.254785 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:47.254954 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.255000 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.255078 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 15:00:47.263262 9746 tablet.cc:2369] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:47.263482 9557 raft_consensus.cc:1261] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 FOLLOWER]: Refusing update from remote peer 1927c10bb1804fbda5f90a3759744e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:47.263484 9698 raft_consensus.cc:1261] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 1 FOLLOWER]: Refusing update from remote peer 1927c10bb1804fbda5f90a3759744e64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:47.264540 9785 consensus_queue.cc:1035] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:47.264819 9788 consensus_queue.cc:1035] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:47.290546 9792 mvcc.cc:205] Tried to move back new op lower bound from 7298939483181015040 to 7298939483056447488. Current Snapshot: MvccSnapshot[applied={T|T < 7298939483181015040 or (T in {7298939483181015040})}]
I20260620 15:00:47.295104 9793 mvcc.cc:205] Tried to move back new op lower bound from 7298939483181015040 to 7298939483056447488. Current Snapshot: MvccSnapshot[applied={T|T < 7298939483181015040}]
I20260620 15:00:47.305341 8486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:47.306064 8486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:47.312673 8486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e355101e7d7451493225e2674944836: Sending DeleteTablet for 3 replicas of tablet cd6d77dc2cdb4844ab2bd1ec371aab7d
I20260620 15:00:47.314015 9678 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:47 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:52856
I20260620 15:00:47.314054 9389 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:47 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:57958
I20260620 15:00:47.314119 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet cd6d77dc2cdb4844ab2bd1ec371aab7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:47 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:32846
I20260620 15:00:47.323830 9804 tablet_replica.cc:330] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: stopping tablet replica
I20260620 15:00:47.324383 9804 raft_consensus.cc:2229] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:47.325093 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 8454
I20260620 15:00:47.326190 8516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:47.327028 9804 raft_consensus.cc:2258] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:47.327755 9803 tablet_replica.cc:330] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: stopping tablet replica
I20260620 15:00:47.328295 9803 raft_consensus.cc:2229] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:47.329185 9803 raft_consensus.cc:2258] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:47.331036 9804 ts_tablet_manager.cc:1905] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:47.331036 9803 ts_tablet_manager.cc:1905] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:47.337195 9802 tablet_replica.cc:330] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: stopping tablet replica
I20260620 15:00:47.338073 9802 raft_consensus.cc:2229] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:47.338615 9802 raft_consensus.cc:2258] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:47.340803 9804 ts_tablet_manager.cc:1918] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:47.340975 9804 log.cc:1196] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/cd6d77dc2cdb4844ab2bd1ec371aab7d
I20260620 15:00:47.341240 9803 ts_tablet_manager.cc:1918] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:47.341599 9804 ts_tablet_manager.cc:1939] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 7c6f288b292f48d9a59f76e4d97503c0: Deleting consensus metadata
I20260620 15:00:47.343622 9802 ts_tablet_manager.cc:1905] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:47.345595 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.317068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.2.70.61:36919)
0620 15:00:47.317277 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.317287 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.324033 (+ 6746us) server_negotiation.cc:407] Connection header received
0620 15:00:47.324309 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.324314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.324406 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.324484 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.328436 (+ 3952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.329804 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.332562 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.333204 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.333302 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.337578 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.337589 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.337597 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.337651 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.342417 (+ 4766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.343372 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.343378 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.343382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.343515 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.343972 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.343977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.343980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.344339 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.344584 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.345008 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.345229 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:00:47.341436 9803 log.cc:1196] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/cd6d77dc2cdb4844ab2bd1ec371aab7d
I20260620 15:00:47.349849 9803 ts_tablet_manager.cc:1939] T cd6d77dc2cdb4844ab2bd1ec371aab7d P 1927c10bb1804fbda5f90a3759744e64: Deleting consensus metadata
I20260620 15:00:47.351964 9802 ts_tablet_manager.cc:1918] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:47.352133 9802 log.cc:1196] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/cd6d77dc2cdb4844ab2bd1ec371aab7d
I20260620 15:00:47.352694 9802 ts_tablet_manager.cc:1939] T cd6d77dc2cdb4844ab2bd1ec371aab7d P e48b32dc05114c65889487a5089f96d3: Deleting consensus metadata
I20260620 15:00:47.360647 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.324510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44314 (local address 127.2.70.60:46775)
0620 15:00:47.324692 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.324701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.345728 (+ 21027us) server_negotiation.cc:407] Connection header received
0620 15:00:47.345976 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.345983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.346074 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.346143 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.348204 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.349279 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.351499 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.352158 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.352288 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.355518 (+ 3230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.355523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.355526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.355559 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.358120 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.358752 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.358757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.358762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.358891 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.359263 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.359268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.359272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.359552 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.359809 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.360102 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.360306 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 15:00:47.488445 8470 catalog_manager.cc:4841] TS 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921): tablet cd6d77dc2cdb4844ab2bd1ec371aab7d (table test-table [id=7e9d8170ef5a4f8dad75596998c27100]) successfully deleted
I20260620 15:00:47.489905 8472 catalog_manager.cc:4841] TS e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307): tablet cd6d77dc2cdb4844ab2bd1ec371aab7d (table test-table [id=7e9d8170ef5a4f8dad75596998c27100]) successfully deleted
I20260620 15:00:47.490329 8473 catalog_manager.cc:4841] TS 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2:36669): tablet cd6d77dc2cdb4844ab2bd1ec371aab7d (table test-table [id=7e9d8170ef5a4f8dad75596998c27100]) successfully deleted
I20260620 15:00:47.490613 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 8454
W20260620 15:00:47.490541 8473 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.2.70.2:36669, user_credentials={real_user=slave}} blocked reactor thread for 147071us
I20260620 15:00:47.547135 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.526127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.2.70.60:46775)
0620 15:00:47.526316 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.526322 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.530588 (+ 4266us) server_negotiation.cc:407] Connection header received
0620 15:00:47.530936 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.530943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.531055 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.531138 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.533168 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.534142 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.536062 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.536684 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.536798 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.539877 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.539883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.539887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.539925 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.543620 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.544520 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.544527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.544531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.544662 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.545114 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.545119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.545122 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.545444 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.545728 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.546348 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.546573 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:00:47.548030 9761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:47.525311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34274 (local address 127.2.70.61:36919)
0620 15:00:47.525934 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:47.525944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:47.530770 (+ 4826us) server_negotiation.cc:407] Connection header received
0620 15:00:47.531304 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:47.531308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:47.531365 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:47.531439 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:47.533808 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.534665 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.535961 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:47.536651 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:47.536783 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:47.540261 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:47.540269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:47.540274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:47.540316 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:47.543828 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.544782 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.544786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.544788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.544905 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:47.545321 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:47.545326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:47.545330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:47.545713 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:47.545948 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:47.547448 (+ 1500us) server_negotiation.cc:299] Negotiation successful
0620 15:00:47.547660 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420}
I20260620 15:00:48.284049 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.283156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.3:40523)
0620 15:00:48.283620 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.283769 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
W20260620 15:00:48.284641 9744 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:40617 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
I20260620 15:00:48.288303 9815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.287216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.1:56583)
0620 15:00:48.287960 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.288138 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":492,"threads_started":1}
W20260620 15:00:48.288633 9455 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:40617 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
I20260620 15:00:48.291807 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.291042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.2:46721)
0620 15:00:48.291447 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.291585 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":206,"threads_started":1}
W20260620 15:00:48.292137 9603 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:40617 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
I20260620 15:00:48.813112 9817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e355101e7d7451493225e2674944836)
I20260620 15:00:48.813402 9817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } }
I20260620 15:00:48.814009 9818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3e355101e7d7451493225e2674944836)
I20260620 15:00:48.814251 9818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } }
I20260620 15:00:48.815107 9817 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e355101e7d7451493225e2674944836 (127.2.70.62:40617), 0caab6fcb3b8497d918f6eee38ac8305 (127.2.70.60:46775)
I20260620 15:00:48.816495 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.815267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45854)
0620 15:00:48.815901 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.816137 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":415,"threads_started":1}
I20260620 15:00:48.825989 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.815307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:46775 (local address 127.0.0.1:44340)
0620 15:00:48.816730 (+ 1423us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.816752 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:48.816914 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:48.817952 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:48.817960 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:48.818007 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:48.818326 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.818341 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.820475 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.820485 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.823572 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.823589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.825220 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.825226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.825424 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.825467 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:00:48.825561 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 15:00:48.825711 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"threads_started":1}
I20260620 15:00:48.826835 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.815581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.2.70.60:46775)
0620 15:00:48.817523 (+ 1942us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:48.817533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:48.817556 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:00:48.817641 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:48.817649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:48.817764 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:48.818490 (+ 726us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:48.818714 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.820271 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.823783 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.825033 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.826210 (+ 1177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.826299 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:48.826348 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 15:00:48.826475 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1782,"thread_start_us":1605,"threads_started":1}
I20260620 15:00:48.827553 9818 leader_election.cc:288] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e355101e7d7451493225e2674944836 (127.2.70.62:40617), 4d3023623215480aa17e5317c6334491 (127.2.70.61:36919)
W20260620 15:00:48.828251 9071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
I20260620 15:00:48.828709 9284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d3023623215480aa17e5317c6334491" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0caab6fcb3b8497d918f6eee38ac8305" is_pre_election: true
I20260620 15:00:48.829073 9284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d3023623215480aa17e5317c6334491 in term 1.
I20260620 15:00:48.829691 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.817456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45862)
0620 15:00:48.828441 (+ 10985us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.829356 (+ 915us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10846,"thread_start_us":9732,"threads_started":1}
I20260620 15:00:48.829672 9069 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0caab6fcb3b8497d918f6eee38ac8305, 4d3023623215480aa17e5317c6334491; no voters:
I20260620 15:00:48.830060 9817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260620 15:00:48.830080 9259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
I20260620 15:00:48.830186 9817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3e355101e7d7451493225e2674944836)
I20260620 15:00:48.830304 9817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:00:48.830506 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.829902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45874)
0620 15:00:48.830144 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.830277 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260620 15:00:48.830704 9071 leader_election.cc:334] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e355101e7d7451493225e2674944836 (127.2.70.62:40617): Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
I20260620 15:00:48.832394 9822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.831989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45882)
0620 15:00:48.832161 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.832248 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:00:48.832607 9259 leader_election.cc:334] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3e355101e7d7451493225e2674944836 (127.2.70.62:40617): Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
I20260620 15:00:48.833410 9817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } }
I20260620 15:00:48.834110 9817 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [CANDIDATE]: Term 2 election: Requested vote from peers 3e355101e7d7451493225e2674944836 (127.2.70.62:40617), 0caab6fcb3b8497d918f6eee38ac8305 (127.2.70.60:46775)
I20260620 15:00:48.834640 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.834179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45886)
0620 15:00:48.834413 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.834478 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260620 15:00:48.834686 9284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d3023623215480aa17e5317c6334491" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0caab6fcb3b8497d918f6eee38ac8305"
I20260620 15:00:48.834893 9284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:00:48.835637 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.835218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45900)
0620 15:00:48.835365 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.835448 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:00:48.835860 9071 leader_election.cc:334] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3e355101e7d7451493225e2674944836 (127.2.70.62:40617): Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
I20260620 15:00:48.836467 9823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.816176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.2.70.61:36919)
0620 15:00:48.829177 (+ 13001us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:48.829183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:48.829204 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:00:48.829259 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:48.829263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:48.829334 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:48.829484 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:48.830936 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.832214 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.834102 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.835464 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.835618 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.835731 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:48.836033 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 15:00:48.836154 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12876,"thread_start_us":11758,"threads_started":1}
I20260620 15:00:48.836526 9824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.815816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:36919 (local address 127.0.0.1:34290)
0620 15:00:48.827495 (+ 11679us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.827517 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:48.827668 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:48.829596 (+ 1928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:48.829606 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:48.829654 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:48.830457 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.830478 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.832351 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.832357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.833947 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.833959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.835657 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.835663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.835859 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.835891 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:00:48.835992 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:00:48.836137 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11535,"thread_start_us":12847,"threads_started":1}
I20260620 15:00:48.837210 9095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0caab6fcb3b8497d918f6eee38ac8305" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4d3023623215480aa17e5317c6334491" is_pre_election: true
I20260620 15:00:48.837558 9095 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0caab6fcb3b8497d918f6eee38ac8305 in current term 2: Already voted for candidate 4d3023623215480aa17e5317c6334491 in this term.
I20260620 15:00:48.837746 9284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d3023623215480aa17e5317c6334491 in term 2.
I20260620 15:00:48.838097 9069 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0caab6fcb3b8497d918f6eee38ac8305, 4d3023623215480aa17e5317c6334491; no voters: 3e355101e7d7451493225e2674944836
I20260620 15:00:48.838174 9260 leader_election.cc:302] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0caab6fcb3b8497d918f6eee38ac8305; no voters: 3e355101e7d7451493225e2674944836, 4d3023623215480aa17e5317c6334491
I20260620 15:00:48.838375 9817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:00:48.838586 9818 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 15:00:48.838889 9817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [term 2 LEADER]: Becoming Leader. State: Replica: 4d3023623215480aa17e5317c6334491, State: Running, Role: LEADER
I20260620 15:00:48.839255 9817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } }
I20260620 15:00:48.840746 9827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3023623215480aa17e5317c6334491. Latest consensus state: current_term: 2 leader_uuid: "4d3023623215480aa17e5317c6334491" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:48.841048 9827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:48.841810 9829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:48.845296 9829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:48.846170 9829 catalog_manager.cc:1259] Loaded cluster ID: bf8a40f7f97a4ad88e5e36747ef37dc7
I20260620 15:00:48.846269 9829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:48.847355 9829 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:48.848137 9829 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Loaded TSK: 0
I20260620 15:00:48.848959 9829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:00:48.883456 9084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:48.890731 9084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:48.894728 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.894335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45910)
0620 15:00:48.894504 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.894598 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 15:00:48.894858 9284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [term 2 FOLLOWER]: Refusing update from remote peer 4d3023623215480aa17e5317c6334491: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 15:00:48.895606 9827 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 15:00:48.895776 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.895367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45918)
0620 15:00:48.895534 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.895639 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:00:48.896044 9071 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 -> Peer 3e355101e7d7451493225e2674944836 (127.2.70.62:40617): Couldn't send request to peer 3e355101e7d7451493225e2674944836. Status: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 15:00:48.898452 9818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3023623215480aa17e5317c6334491. Latest consensus state: current_term: 2 leader_uuid: "4d3023623215480aa17e5317c6334491" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:48.898711 9818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:48.899978 9817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4d3023623215480aa17e5317c6334491" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:48.900312 9817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:48.900774 9818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4d3023623215480aa17e5317c6334491" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3e355101e7d7451493225e2674944836" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40617 } } peers { permanent_uuid: "4d3023623215480aa17e5317c6334491" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 36919 } attrs { promote: false } } peers { permanent_uuid: "0caab6fcb3b8497d918f6eee38ac8305" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 46775 } attrs { promote: false } } }
I20260620 15:00:48.901028 9818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caab6fcb3b8497d918f6eee38ac8305 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:00:48.918056 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.908945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:36669 (local address 127.0.0.1:32856)
0620 15:00:48.909073 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.909085 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:48.909147 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:48.910012 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:48.910017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:48.910040 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:48.910223 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.910250 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.911746 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.911755 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.914384 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.914416 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.916864 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.916874 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.917180 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.917239 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 15:00:48.917322 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 15:00:48.917569 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 15:00:48.919344 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.909279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.2.70.2:36669)
0620 15:00:48.909751 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:48.909755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:48.909771 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:00:48.909825 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:48.909828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:48.909880 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:48.909989 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:48.910414 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.911520 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.914960 (+ 3440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.916608 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.918765 (+ 2157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.918869 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:48.918922 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 15:00:48.919068 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":289,"threads_started":1}
I20260620 15:00:48.920228 9537 tablet_service.cc:1467] Processing CreateTablet for tablet 430b49ab277f4557a99394705a858f98 (DEFAULT_TABLE table=test-table [id=2b4c0013a3d24201b5f930edfbfbebbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:48.920804 9537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430b49ab277f4557a99394705a858f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:48.931075 9840 tablet_bootstrap.cc:492] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: Bootstrap starting.
I20260620 15:00:48.931159 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.910657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.2.70.1:40921)
0620 15:00:48.919965 (+ 9308us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:48.919971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:48.920352 (+ 381us) server_negotiation.cc:407] Connection header received
0620 15:00:48.920564 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:48.920570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:48.920655 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:48.921190 (+ 535us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:48.921790 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.923239 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.925988 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.927688 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.928321 (+ 633us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.928468 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:48.930769 (+ 2301us) server_negotiation.cc:299] Negotiation successful
0620 15:00:48.930895 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9183,"thread_start_us":9076,"threads_started":1}
I20260620 15:00:48.931254 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.910472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40921 (local address 127.0.0.1:57976)
0620 15:00:48.918370 (+ 7898us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.918388 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:48.918488 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:48.921175 (+ 2687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:48.921184 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:48.921222 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:48.921507 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.921519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.923406 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.923411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.925736 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.925757 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.927906 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.927915 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.928238 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.930641 (+ 2403us) client_negotiation.cc:769] Sending connection context
0620 15:00:48.930724 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 15:00:48.930891 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7773,"mutex_wait_us":2297,"spinlock_wait_cycles":76160,"thread_start_us":19994,"threads_started":1}
I20260620 15:00:48.931995 9389 tablet_service.cc:1467] Processing CreateTablet for tablet 430b49ab277f4557a99394705a858f98 (DEFAULT_TABLE table=test-table [id=2b4c0013a3d24201b5f930edfbfbebbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:48.932394 9389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430b49ab277f4557a99394705a858f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:48.933619 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.908579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:39307 (local address 127.0.0.1:52866)
0620 15:00:48.908735 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.908754 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:48.908882 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:48.926039 (+ 17157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:48.926045 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:48.926072 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:48.926338 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.926348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.927839 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.927846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.930410 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:48.930433 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.932874 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.932882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:48.933091 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.933128 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:00:48.933191 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 15:00:48.933312 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 15:00:48.933805 9840 tablet_bootstrap.cc:654] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:48.934460 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.908684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.2.70.3:39307)
0620 15:00:48.925552 (+ 16868us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:48.925558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:48.925578 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:00:48.925645 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:48.925653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:48.925732 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:48.926483 (+ 751us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:48.926653 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.927694 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.931507 (+ 3813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:48.932632 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:48.933840 (+ 1208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:48.933966 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:48.934025 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 15:00:48.934174 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16714,"thread_start_us":17141,"threads_started":1}
I20260620 15:00:48.937177 9842 tablet_bootstrap.cc:492] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: Bootstrap starting.
I20260620 15:00:48.937381 9678 tablet_service.cc:1467] Processing CreateTablet for tablet 430b49ab277f4557a99394705a858f98 (DEFAULT_TABLE table=test-table [id=2b4c0013a3d24201b5f930edfbfbebbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:48.937832 9678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430b49ab277f4557a99394705a858f98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:48.939388 9842 tablet_bootstrap.cc:654] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:48.941349 9840 tablet_bootstrap.cc:492] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: No bootstrap required, opened a new log
I20260620 15:00:48.941526 9840 ts_tablet_manager.cc:1397] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260620 15:00:48.942183 9842 tablet_bootstrap.cc:492] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: No bootstrap required, opened a new log
I20260620 15:00:48.942332 9840 raft_consensus.cc:348] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:48.942579 9840 raft_consensus.cc:374] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:48.942667 9840 raft_consensus.cc:729] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6f288b292f48d9a59f76e4d97503c0, State: Initialized, Role: FOLLOWER
I20260620 15:00:48.942703 9842 ts_tablet_manager.cc:1397] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260620 15:00:48.942941 9840 consensus_queue.cc:260] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:48.943426 9844 tablet_bootstrap.cc:492] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: Bootstrap starting.
I20260620 15:00:48.943444 9842 raft_consensus.cc:348] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:48.943636 9842 raft_consensus.cc:374] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:48.943732 9842 raft_consensus.cc:729] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1927c10bb1804fbda5f90a3759744e64, State: Initialized, Role: FOLLOWER
I20260620 15:00:48.943938 9842 consensus_queue.cc:260] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:48.945710 9844 tablet_bootstrap.cc:654] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:48.947237 9840 ts_tablet_manager.cc:1428] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 15:00:48.950374 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.947511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.2:60927)
0620 15:00:48.950063 (+ 2552us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.950171 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2423,"thread_start_us":2278,"threads_started":1}
I20260620 15:00:48.951735 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.951207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.2:42179)
0620 15:00:48.951468 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.951564 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 15:00:48.952713 9604 heartbeater.cc:503] Master 127.2.70.61:36919 was elected leader, sending a full tablet report...
I20260620 15:00:48.952740 9842 ts_tablet_manager.cc:1428] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260620 15:00:48.954238 9844 tablet_bootstrap.cc:492] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: No bootstrap required, opened a new log
I20260620 15:00:48.954411 9844 ts_tablet_manager.cc:1397] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20260620 15:00:48.955471 9844 raft_consensus.cc:348] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:48.955718 9844 raft_consensus.cc:374] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:48.955821 9844 raft_consensus.cc:729] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48b32dc05114c65889487a5089f96d3, State: Initialized, Role: FOLLOWER
I20260620 15:00:48.956166 9844 consensus_queue.cc:260] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:48.957013 9844 ts_tablet_manager.cc:1428] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 15:00:48.957026 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.953335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.1:39691)
0620 15:00:48.956645 (+ 3310us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.956775 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3161,"thread_start_us":3050,"threads_started":1}
I20260620 15:00:48.959504 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.957567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.3:57803)
0620 15:00:48.959056 (+ 1489us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.959192 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1342,"thread_start_us":1218,"threads_started":1}
I20260620 15:00:48.959693 9745 heartbeater.cc:503] Master 127.2.70.61:36919 was elected leader, sending a full tablet report...
I20260620 15:00:48.960592 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:48.960275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.3:60621)
0620 15:00:48.960422 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:48.960496 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:00:49.029055 9457 tablet.cc:2369] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:49.030627 9845 raft_consensus.cc:482] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:49.030818 9845 raft_consensus.cc:504] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:49.032450 9845 leader_election.cc:288] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307), 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921)
I20260620 15:00:49.038041 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.032823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:47989 (local address 127.2.70.3:39307)
0620 15:00:49.033014 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:49.033020 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:49.033042 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:00:49.033199 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:49.033204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:49.033285 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:49.033391 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:49.033948 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.034661 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.036318 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.037086 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.037232 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:49.037336 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:49.037642 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 15:00:49.037766 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:00:49.038089 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.032646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:39307 (local address 127.2.70.2:47989)
0620 15:00:49.032901 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.032934 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:49.033071 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:49.033438 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:49.033446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:49.033472 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:49.033798 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:49.033807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.034789 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.034796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:49.036160 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:49.036172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.037266 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.037272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:49.037441 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:49.037465 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:00:49.037619 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 15:00:49.037758 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":1}
I20260620 15:00:49.038779 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.032862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:35261 (local address 127.2.70.1:40921)
0620 15:00:49.033044 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:49.033049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:49.033752 (+ 703us) server_negotiation.cc:407] Connection header received
0620 15:00:49.033950 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:49.033958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:49.034042 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:49.034142 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:49.034657 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.035335 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.036913 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.037770 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.037912 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:49.038018 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:49.038457 (+ 439us) server_negotiation.cc:299] Negotiation successful
0620 15:00:49.038611 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:00:49.038765 9698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430b49ab277f4557a99394705a858f98" candidate_uuid: "7c6f288b292f48d9a59f76e4d97503c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48b32dc05114c65889487a5089f96d3" is_pre_election: true
I20260620 15:00:49.038957 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.032646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40921 (local address 127.2.70.2:35261)
0620 15:00:49.033654 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.033668 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:49.033811 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:49.034157 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:49.034161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:49.034183 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:49.034514 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:49.034523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.035473 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.035481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:49.036775 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:49.036785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.037983 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.037989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:49.038230 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:49.038272 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 15:00:49.038445 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 15:00:49.038567 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":415,"thread_start_us":317,"threads_started":1}
I20260620 15:00:49.039026 9698 raft_consensus.cc:2454] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6f288b292f48d9a59f76e4d97503c0 in term 0.
I20260620 15:00:49.039557 9492 leader_election.cc:302] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6f288b292f48d9a59f76e4d97503c0, e48b32dc05114c65889487a5089f96d3; no voters:
I20260620 15:00:49.039742 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430b49ab277f4557a99394705a858f98" candidate_uuid: "7c6f288b292f48d9a59f76e4d97503c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1927c10bb1804fbda5f90a3759744e64" is_pre_election: true
I20260620 15:00:49.039916 9845 raft_consensus.cc:2781] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:49.040053 9845 raft_consensus.cc:482] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:49.040143 9845 raft_consensus.cc:3037] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:49.040124 9409 raft_consensus.cc:2454] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6f288b292f48d9a59f76e4d97503c0 in term 0.
I20260620 15:00:49.042541 9845 raft_consensus.cc:504] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:49.043167 9845 leader_election.cc:288] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [CANDIDATE]: Term 1 election: Requested vote from peers e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307), 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921)
I20260620 15:00:49.043524 9698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430b49ab277f4557a99394705a858f98" candidate_uuid: "7c6f288b292f48d9a59f76e4d97503c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48b32dc05114c65889487a5089f96d3"
I20260620 15:00:49.043737 9698 raft_consensus.cc:3037] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:49.043795 9409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430b49ab277f4557a99394705a858f98" candidate_uuid: "7c6f288b292f48d9a59f76e4d97503c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1927c10bb1804fbda5f90a3759744e64"
I20260620 15:00:49.043942 9409 raft_consensus.cc:3037] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:49.045733 9698 raft_consensus.cc:2454] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6f288b292f48d9a59f76e4d97503c0 in term 1.
I20260620 15:00:49.045945 9409 raft_consensus.cc:2454] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6f288b292f48d9a59f76e4d97503c0 in term 1.
I20260620 15:00:49.046130 9492 leader_election.cc:302] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c6f288b292f48d9a59f76e4d97503c0, e48b32dc05114c65889487a5089f96d3; no voters:
I20260620 15:00:49.046411 9845 raft_consensus.cc:2781] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:49.046864 9845 raft_consensus.cc:686] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 LEADER]: Becoming Leader. State: Replica: 7c6f288b292f48d9a59f76e4d97503c0, State: Running, Role: LEADER
I20260620 15:00:49.047453 9845 consensus_queue.cc:237] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } }
I20260620 15:00:49.047852 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.047394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.2:53675)
0620 15:00:49.047546 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.047694 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:00:49.050908 9084 catalog_manager.cc:5495] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2). New cstate: current_term: 1 leader_uuid: "7c6f288b292f48d9a59f76e4d97503c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c6f288b292f48d9a59f76e4d97503c0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:49.074383 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.069131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.2.70.2:36669)
0620 15:00:49.069367 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:49.069373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:49.069391 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:00:49.069643 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:49.069647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:49.069724 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:49.069878 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:49.070226 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.070915 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.072168 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.072584 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.072718 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:49.073133 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:49.073816 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:49.073968 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:49.074032 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 15:00:49.074117 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:00:49.078751 9698 raft_consensus.cc:1261] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 1 FOLLOWER]: Refusing update from remote peer 7c6f288b292f48d9a59f76e4d97503c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:49.079130 9409 raft_consensus.cc:1261] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 1 FOLLOWER]: Refusing update from remote peer 7c6f288b292f48d9a59f76e4d97503c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:49.079517 9853 consensus_queue.cc:1035] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48b32dc05114c65889487a5089f96d3" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:49.079946 9845 consensus_queue.cc:1035] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1927c10bb1804fbda5f90a3759744e64" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:49.082154 9456 heartbeater.cc:503] Master 127.2.70.61:36919 was elected leader, sending a full tablet report...
I20260620 15:00:49.082197 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.081788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.1:45271)
0620 15:00:49.081937 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.082029 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 15:00:49.083139 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.082706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.3:38741)
0620 15:00:49.082841 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.082966 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 15:00:49.089196 9849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.088716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.1:59683)
0620 15:00:49.088898 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.089004 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 15:00:49.090687 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.090068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.3:41077)
0620 15:00:49.090318 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.090484 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260620 15:00:49.092432 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.091967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.2:57405)
0620 15:00:49.092187 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.092273 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260620 15:00:49.096513 9852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.095896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.2.70.2:38443)
0620 15:00:49.096158 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.096290 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260620 15:00:49.101997 9085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:49.102284 9085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:49.109894 9085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d3023623215480aa17e5317c6334491: Sending DeleteTablet for 3 replicas of tablet 430b49ab277f4557a99394705a858f98
I20260620 15:00:49.110816 9678 tablet_service.cc:1514] Processing DeleteTablet for tablet 430b49ab277f4557a99394705a858f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:49 UTC) from {username='oryx', principal='oryx/127.2.70.61@KRBTEST.COM'} at 127.0.0.1:52866
I20260620 15:00:49.110942 9537 tablet_service.cc:1514] Processing DeleteTablet for tablet 430b49ab277f4557a99394705a858f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:49 UTC) from {username='oryx', principal='oryx/127.2.70.61@KRBTEST.COM'} at 127.0.0.1:32856
I20260620 15:00:49.112725 9677 tablet_service.cc:1514] Processing DeleteTablet for tablet 430b49ab277f4557a99394705a858f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:49 UTC) from {username='oryx', principal='oryx/127.2.70.61@KRBTEST.COM'} at 127.0.0.1:52866
I20260620 15:00:49.113205 9389 tablet_service.cc:1514] Processing DeleteTablet for tablet 430b49ab277f4557a99394705a858f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:49 UTC) from {username='oryx', principal='oryx/127.2.70.61@KRBTEST.COM'} at 127.0.0.1:57976
I20260620 15:00:49.113224 9388 tablet_service.cc:1514] Processing DeleteTablet for tablet 430b49ab277f4557a99394705a858f98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:49 UTC) from {username='oryx', principal='oryx/127.2.70.61@KRBTEST.COM'} at 127.0.0.1:57976
I20260620 15:00:49.113884 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 9261
I20260620 15:00:49.117962 9868 tablet_replica.cc:330] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: stopping tablet replica
I20260620 15:00:49.118161 9868 raft_consensus.cc:2229] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:49.118474 9868 raft_consensus.cc:2258] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:49.119043 9866 tablet_replica.cc:330] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: stopping tablet replica
I20260620 15:00:49.119232 9866 raft_consensus.cc:2229] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:49.119505 9866 raft_consensus.cc:2258] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:49.120213 9867 tablet_replica.cc:330] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: stopping tablet replica
I20260620 15:00:49.120406 9867 raft_consensus.cc:2229] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:49.121160 9867 raft_consensus.cc:2258] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:49.121232 9866 ts_tablet_manager.cc:1905] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:49.122242 9867 ts_tablet_manager.cc:1905] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:49.129253 9868 ts_tablet_manager.cc:1905] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:49.131057 9866 ts_tablet_manager.cc:1918] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:49.131289 9866 log.cc:1196] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/430b49ab277f4557a99394705a858f98
I20260620 15:00:49.131964 9866 ts_tablet_manager.cc:1939] T 430b49ab277f4557a99394705a858f98 P e48b32dc05114c65889487a5089f96d3: Deleting consensus metadata
I20260620 15:00:49.133666 9071 catalog_manager.cc:4841] TS e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307): tablet 430b49ab277f4557a99394705a858f98 (table test-table [id=2b4c0013a3d24201b5f930edfbfbebbe]) successfully deleted
W20260620 15:00:49.135030 9071 catalog_manager.cc:4808] TS e48b32dc05114c65889487a5089f96d3 (127.2.70.3:39307): delete failed for tablet 430b49ab277f4557a99394705a858f98 because the tablet was not found. No further retry: Not found: Tablet not found: 430b49ab277f4557a99394705a858f98
I20260620 15:00:49.135854 9868 ts_tablet_manager.cc:1918] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:49.136006 9868 log.cc:1196] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/430b49ab277f4557a99394705a858f98
I20260620 15:00:49.136467 9868 ts_tablet_manager.cc:1939] T 430b49ab277f4557a99394705a858f98 P 1927c10bb1804fbda5f90a3759744e64: Deleting consensus metadata
I20260620 15:00:49.137663 9069 catalog_manager.cc:4841] TS 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921): tablet 430b49ab277f4557a99394705a858f98 (table test-table [id=2b4c0013a3d24201b5f930edfbfbebbe]) successfully deleted
I20260620 15:00:49.139278 9867 ts_tablet_manager.cc:1918] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:49.139433 9867 log.cc:1196] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/430b49ab277f4557a99394705a858f98
I20260620 15:00:49.139999 9867 ts_tablet_manager.cc:1939] T 430b49ab277f4557a99394705a858f98 P 7c6f288b292f48d9a59f76e4d97503c0: Deleting consensus metadata
I20260620 15:00:49.141134 9072 catalog_manager.cc:4841] TS 7c6f288b292f48d9a59f76e4d97503c0 (127.2.70.2:36669): tablet 430b49ab277f4557a99394705a858f98 (table test-table [id=2b4c0013a3d24201b5f930edfbfbebbe]) successfully deleted
W20260620 15:00:49.141707 9069 catalog_manager.cc:4808] TS 1927c10bb1804fbda5f90a3759744e64 (127.2.70.1:40921): delete failed for tablet 430b49ab277f4557a99394705a858f98 because the tablet was not found. No further retry: Not found: Tablet not found: 430b49ab277f4557a99394705a858f98
I20260620 15:00:49.151033 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.114629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.2.70.1:40921)
0620 15:00:49.114801 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:49.114808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:49.138018 (+ 23210us) server_negotiation.cc:407] Connection header received
0620 15:00:49.138191 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:49.138194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:49.138255 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:49.138373 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:49.139725 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.140544 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.142311 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.142794 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.142932 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:49.145430 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:49.145445 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:49.145465 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:49.145557 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:49.148291 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:49.148924 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:49.148933 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:49.148937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:49.149059 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:49.149428 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:49.149435 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:49.149440 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:49.149873 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:49.150182 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:49.150441 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 15:00:49.150648 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:00:49.151625 9449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:49.269596 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 9261
I20260620 15:00:49.300819 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 9466
I20260620 15:00:49.316891 9837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.301646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.2.70.2:36669)
0620 15:00:49.301825 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:49.301832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:49.301887 (+ 55us) server_negotiation.cc:407] Connection header received
0620 15:00:49.302091 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:49.302095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:49.302159 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:49.302247 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:49.303944 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.304782 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.306550 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.306971 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.307091 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:49.310557 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:49.310568 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:49.310583 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:49.310616 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:49.313890 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:49.314736 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:49.314743 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:49.314746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:49.314901 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:49.315350 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:49.315357 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:49.315360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:49.315809 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:49.316045 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:49.316376 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 15:00:49.316546 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:00:49.317621 9597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:49.339757 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.339341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45932)
0620 15:00:49.339509 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.339601 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:00:49.340760 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.340428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:40617 (local address 127.0.0.1:45936)
0620 15:00:49.340574 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:49.340648 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:00:49.436492 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 9466
I20260620 15:00:49.455832 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 9615
I20260620 15:00:49.469223 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:49.456673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.2.70.3:39307)
0620 15:00:49.456908 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:49.456912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:49.456933 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:00:49.456991 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:49.456995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:49.457066 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:49.457152 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:49.458511 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.459276 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.460457 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:49.460836 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:49.460941 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:49.463453 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:49.463468 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:49.463485 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:49.463519 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:49.466385 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:49.467138 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:49.467146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:49.467149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:49.467280 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:49.467694 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:49.467698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:49.467700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:49.468068 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:49.468310 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:49.468659 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 15:00:49.468825 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 15:00:49.470019 9738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:49.565016 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 9615
I20260620 15:00:49.578819 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 8947
I20260620 15:00:49.580008 9115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:49.701650 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 8947
I20260620 15:00:49.731330 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 9047
I20260620 15:00:49.732486 9304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:49.848877 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 9047
2026-06-20T15:00:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9517 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:49 dist-test-slave-n5vv krb5kdc[9877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:49 dist-test-slave-n5vv krb5kdc[9877](info): set up 2 sockets
Jun 20 15:00:49 dist-test-slave-n5vv krb5kdc[9877](info): commencing operation
krb5kdc: starting...
W20260620 15:00:50.946453 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:51 dist-test-slave-n5vv krb5kdc[9877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:51Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.62.keytab.
Entry for principal oryx/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.62.keytab.
I20260620 15:00:51.163475 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43579
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.62.keytab
--principal=oryx/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:43579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:51.483209 9890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:51.483505 9890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:51.483669 9890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:51.491212 9890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:51.491326 9890 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 15:00:51.491410 9890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:51.491500 9890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:51.491575 9890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:51.491676 9890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:51.501662 9890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:43579
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.62.keytab
--principal=oryx/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:51.502976 9890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:51.504312 9890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:51.516356 9895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:51.516321 9898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:51.516407 9896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:51.517261 9890 server_base.cc:1034] running on GCE node
I20260620 15:00:51.518075 9890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:51.519466 9890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:51.520716 9890 hybrid_clock.cc:648] HybridClock initialized: now 1781967651520655 us; error 41 us; skew 500 ppm
I20260620 15:00:51.521062 9890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:51 dist-test-slave-n5vv krb5kdc[9877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967651, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:51.524262 9890 init.cc:377] Logged in from keytab as oryx/127.2.70.62@KRBTEST.COM (short username oryx)
I20260620 15:00:51.525732 9890 webserver.cc:460] Webserver started at http://127.2.70.62:33315/ using document root <none> and password file <none>
I20260620 15:00:51.526386 9890 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:51.526466 9890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:51.526774 9890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:51.529706 9890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "f550186d971949a980cd1ef4256f2004"
format_stamp: "Formatted at 2026-06-20 15:00:51 on dist-test-slave-n5vv"
server_key: "ab10d97f99755f31314224de60fc611f"
server_key_iv: "e3324493c3bf977a968bb579440e5f59"
server_key_version: "encryptionkey@0"
I20260620 15:00:51.530512 9890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "f550186d971949a980cd1ef4256f2004"
format_stamp: "Formatted at 2026-06-20 15:00:51 on dist-test-slave-n5vv"
server_key: "ab10d97f99755f31314224de60fc611f"
server_key_iv: "e3324493c3bf977a968bb579440e5f59"
server_key_version: "encryptionkey@0"
I20260620 15:00:51.535599 9890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:00:51.539012 9905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:51.540580 9890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:00:51.540760 9890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "f550186d971949a980cd1ef4256f2004"
format_stamp: "Formatted at 2026-06-20 15:00:51 on dist-test-slave-n5vv"
server_key: "ab10d97f99755f31314224de60fc611f"
server_key_iv: "e3324493c3bf977a968bb579440e5f59"
server_key_version: "encryptionkey@0"
I20260620 15:00:51.540979 9890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:51.559266 9890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:51.563680 9890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:51.563962 9890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:51.585032 9890 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:43579
I20260620 15:00:51.585068 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:43579 every 8 connection(s)
I20260620 15:00:51.586485 9890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:51.590488 9967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:51.593416 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 9890
I20260620 15:00:51.593672 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:51.594312 2328 external_mini_cluster.cc:1428] Setting key 813af355b35f751b1b680ef44ad64b35
I20260620 15:00:51.603917 9967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: Bootstrap starting.
I20260620 15:00:51.607504 9967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:51.608515 9967 log.cc:824] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:00:51 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:51.611747 9967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: No bootstrap required, opened a new log
I20260620 15:00:51.619194 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:51.596398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55664 (local address 127.2.70.62:43579)
0620 15:00:51.597205 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:51.597220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:51.597270 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:00:51.599560 (+ 2290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:51.599596 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:51.600144 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:51.600625 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:51.601941 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:51.603865 (+ 1924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:51.605148 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:51.606284 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:51.606462 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:51.610674 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:51.610684 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:51.610703 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:51.610755 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:51.614648 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:51.615305 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:51.615314 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:51.615322 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:51.615545 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:51.615894 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:51.615899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:51.615903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:51.616617 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:51.616892 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:51.617258 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 15:00:51.617809 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":341,"threads_started":1}
I20260620 15:00:51.621901 9967 raft_consensus.cc:348] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f550186d971949a980cd1ef4256f2004" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43579 } }
I20260620 15:00:51.622257 9967 raft_consensus.cc:374] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:51.622378 9967 raft_consensus.cc:729] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f550186d971949a980cd1ef4256f2004, State: Initialized, Role: FOLLOWER
I20260620 15:00:51.622900 9967 consensus_queue.cc:260] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f550186d971949a980cd1ef4256f2004" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43579 } }
I20260620 15:00:51.623144 9967 raft_consensus.cc:388] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:51.623272 9967 raft_consensus.cc:482] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:51.623418 9967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:51.625567 9967 raft_consensus.cc:504] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f550186d971949a980cd1ef4256f2004" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43579 } }
I20260620 15:00:51.626077 9967 leader_election.cc:302] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f550186d971949a980cd1ef4256f2004; no voters:
I20260620 15:00:51.626629 9967 leader_election.cc:288] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:51.626749 9972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:51.627995 9972 raft_consensus.cc:686] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [term 1 LEADER]: Becoming Leader. State: Replica: f550186d971949a980cd1ef4256f2004, State: Running, Role: LEADER
I20260620 15:00:51.628604 9972 consensus_queue.cc:237] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f550186d971949a980cd1ef4256f2004" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43579 } }
I20260620 15:00:51.628957 9967 sys_catalog.cc:564] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:51.631500 9973 sys_catalog.cc:455] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f550186d971949a980cd1ef4256f2004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f550186d971949a980cd1ef4256f2004" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43579 } } }
I20260620 15:00:51.631517 9974 sys_catalog.cc:455] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f550186d971949a980cd1ef4256f2004. Latest consensus state: current_term: 1 leader_uuid: "f550186d971949a980cd1ef4256f2004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f550186d971949a980cd1ef4256f2004" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43579 } } }
I20260620 15:00:51.631960 9974 sys_catalog.cc:458] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:51.631958 9973 sys_catalog.cc:458] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004 [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:51.638204 9978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:51.644106 9978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:51.653834 9978 catalog_manager.cc:1347] Generated new cluster ID: 6b044bf1f266484483a2ea839d17a7e3
I20260620 15:00:51.654139 9978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:51.691609 9978 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:51.693606 9978 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:51.715468 9978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: Generated new TSK 0
I20260620 15:00:51.716454 9978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab.
Entry for principal oryx/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab.
I20260620 15:00:51.779050 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43579
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab
--principal=oryx/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:52.113608 9993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:52.113883 9993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:52.114050 9993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:52.121304 9993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:52.121429 9993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:52.121587 9993 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:00:52.121719 9993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:52.131956 9993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab
--principal=oryx/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:52.133261 9993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:52.134716 9993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:52.148689 10001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:52.148756 9998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:52.148716 9999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:52.148983 9993 server_base.cc:1034] running on GCE node
I20260620 15:00:52.149938 9993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:52.150861 9993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:52.152122 9993 hybrid_clock.cc:648] HybridClock initialized: now 1781967652152048 us; error 49 us; skew 500 ppm
I20260620 15:00:52.152438 9993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:52 dist-test-slave-n5vv krb5kdc[9877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967652, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:52.155589 9993 init.cc:377] Logged in from keytab as oryx/127.2.70.1@KRBTEST.COM (short username oryx)
I20260620 15:00:52.156948 9993 webserver.cc:460] Webserver started at http://127.2.70.1:41259/ using document root <none> and password file <none>
I20260620 15:00:52.157698 9993 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:52.157829 9993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:52.158154 9993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:52.161032 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "378410d6e2894f46848b87c1c20bd3b2"
format_stamp: "Formatted at 2026-06-20 15:00:52 on dist-test-slave-n5vv"
server_key: "bbcb4f02c9ed2148ef9f1d4f6d604c12"
server_key_iv: "a95e96a0f7356ec14c2d1557c0cb668d"
server_key_version: "encryptionkey@0"
I20260620 15:00:52.161857 9993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "378410d6e2894f46848b87c1c20bd3b2"
format_stamp: "Formatted at 2026-06-20 15:00:52 on dist-test-slave-n5vv"
server_key: "bbcb4f02c9ed2148ef9f1d4f6d604c12"
server_key_iv: "a95e96a0f7356ec14c2d1557c0cb668d"
server_key_version: "encryptionkey@0"
I20260620 15:00:52.166879 9993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:00:52.170404 10008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:52.171944 9993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:52.172112 9993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "378410d6e2894f46848b87c1c20bd3b2"
format_stamp: "Formatted at 2026-06-20 15:00:52 on dist-test-slave-n5vv"
server_key: "bbcb4f02c9ed2148ef9f1d4f6d604c12"
server_key_iv: "a95e96a0f7356ec14c2d1557c0cb668d"
server_key_version: "encryptionkey@0"
I20260620 15:00:52.172333 9993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:52.189333 9993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:52.193444 9993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:52.193750 9993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:52.196388 9993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:52.196498 9993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:52.196662 9993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:52.196764 9993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:00:52 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967652, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:52.224434 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:52.197008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:50367 (local address 127.2.70.62:43579)
0620 15:00:52.197666 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:52.197672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:52.198915 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 15:00:52.202256 (+ 3341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:52.202264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:52.202384 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:52.202497 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:52.205762 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.206837 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.210390 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.210871 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.210992 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:52.216393 (+ 5401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:52.216399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:52.216403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:52.216441 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:52.220677 (+ 4236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:52.221832 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:52.221838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:52.221842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:52.222037 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:52.222511 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:52.222516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:52.222520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:52.222832 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:52.223011 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:52.223777 (+ 766us) server_negotiation.cc:299] Negotiation successful
0620 15:00:52.224040 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":393,"threads_started":1}
I20260620 15:00:52.225685 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:52.197760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43579 (local address 127.2.70.1:50367)
0620 15:00:52.198699 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:52.198774 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:52.201844 (+ 3070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:52.203178 (+ 1334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:52.203190 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:52.204223 (+ 1033us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:52.205468 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:52.205530 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.207067 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.207075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:52.210190 (+ 3115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:52.210214 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.211156 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.211166 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:52.211581 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:52.212585 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:52.212637 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:52.216091 (+ 3454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:52.220859 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:52.220877 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:52.220896 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:52.221668 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:52.222151 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:52.222157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:52.222161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:52.222380 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:52.222972 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:52.222982 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:52.223386 (+ 404us) client_negotiation.cc:769] Sending connection context
0620 15:00:52.223679 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 15:00:52.224268 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":382,"threads_started":1}
I20260620 15:00:52.250198 9993 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:38957
I20260620 15:00:52.250248 10128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:38957 every 8 connection(s)
I20260620 15:00:52.250414 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:52.234489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:34127 (local address 127.2.70.62:43579)
0620 15:00:52.234680 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:52.234689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:52.235615 (+ 926us) server_negotiation.cc:407] Connection header received
0620 15:00:52.235878 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:52.235887 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:52.235981 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:52.236098 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:52.237631 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.238929 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.240599 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.241009 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.241157 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:52.243931 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:52.243940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:52.243946 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:52.243989 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:52.247154 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:52.247937 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:52.247945 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:52.247951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:52.248126 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:52.248507 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:52.248512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:52.248515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:52.248868 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:52.249015 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:52.249766 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 15:00:52.250015 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 15:00:52.250741 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:52.234391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43579 (local address 127.2.70.1:34127)
0620 15:00:52.235457 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:52.235487 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:52.235690 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:52.236149 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:52.236160 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:52.236949 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:52.237433 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:52.237451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.239064 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.239068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:52.240446 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:52.240458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.241203 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.241209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:52.241436 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:52.242299 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:52.242345 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:52.243679 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:52.247289 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:52.247299 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:52.247305 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:52.247817 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:52.248265 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:52.248269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:52.248273 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:52.248392 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:52.249011 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:52.249016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:52.249398 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 15:00:52.249648 (+ 250us) client_negotiation.cc:240] Negotiation successful
0620 15:00:52.250100 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":673,"threads_started":1}
I20260620 15:00:52.251857 9993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:52.253059 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 9993
I20260620 15:00:52.253309 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:52.253957 10129 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43579
I20260620 15:00:52.254248 10129 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:52.255021 2328 external_mini_cluster.cc:1428] Setting key 91e16528e3c70b62c5b53765474a6638
I20260620 15:00:52.255143 10129 heartbeater.cc:511] Master 127.2.70.62:43579 requested a full tablet report, sending...
I20260620 15:00:52.257611 9921 ts_manager.cc:194] Registered new tserver with Master: 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957)
I20260620 15:00:52.259836 9921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:50367
WARNING: no policy specified for oryx/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab.
Entry for principal oryx/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab.
I20260620 15:00:52.304450 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43579
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab
--principal=oryx/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:52.629086 10133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:52.629526 10133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:52.629716 10133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:52.637106 10133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:52.637261 10133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:52.637425 10133 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:00:52.637594 10133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:52.647737 10133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab
--principal=oryx/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:52.649032 10133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:52.650439 10133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:52.663491 10139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:52.663558 10138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:52.664089 10141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:52.664199 10133 server_base.cc:1034] running on GCE node
I20260620 15:00:52.664829 10133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:52.665752 10133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:52.666960 10133 hybrid_clock.cc:648] HybridClock initialized: now 1781967652666873 us; error 62 us; skew 500 ppm
I20260620 15:00:52.667268 10133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:52 dist-test-slave-n5vv krb5kdc[9877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967652, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:52.670434 10133 init.cc:377] Logged in from keytab as oryx/127.2.70.2@KRBTEST.COM (short username oryx)
I20260620 15:00:52.671932 10133 webserver.cc:460] Webserver started at http://127.2.70.2:42431/ using document root <none> and password file <none>
I20260620 15:00:52.672636 10133 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:52.672752 10133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:52.673085 10133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:52.676187 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
format_stamp: "Formatted at 2026-06-20 15:00:52 on dist-test-slave-n5vv"
server_key: "840ad10e562eb12e5d22b211e4bc5920"
server_key_iv: "11809c64e99cc7d367316d7f38b16f6b"
server_key_version: "encryptionkey@0"
I20260620 15:00:52.677009 10133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
format_stamp: "Formatted at 2026-06-20 15:00:52 on dist-test-slave-n5vv"
server_key: "840ad10e562eb12e5d22b211e4bc5920"
server_key_iv: "11809c64e99cc7d367316d7f38b16f6b"
server_key_version: "encryptionkey@0"
I20260620 15:00:52.682147 10133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 15:00:52.685654 10148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:52.687060 10133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:52.687223 10133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
format_stamp: "Formatted at 2026-06-20 15:00:52 on dist-test-slave-n5vv"
server_key: "840ad10e562eb12e5d22b211e4bc5920"
server_key_iv: "11809c64e99cc7d367316d7f38b16f6b"
server_key_version: "encryptionkey@0"
I20260620 15:00:52.687422 10133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:52.707828 10133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:52.711860 10133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:52.712150 10133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:52.714999 10133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:52.715116 10133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:52.715284 10133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:52.715359 10133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:00:52 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967652, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:52.742590 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:52.715288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:44807 (local address 127.2.70.62:43579)
0620 15:00:52.715482 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:52.715489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:52.717077 (+ 1588us) server_negotiation.cc:407] Connection header received
0620 15:00:52.719468 (+ 2391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:52.719474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:52.719575 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:52.719691 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:52.722666 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.723705 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.727189 (+ 3484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.727834 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.727983 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:52.733540 (+ 5557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:52.733552 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:52.733559 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:52.733608 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:52.737390 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:52.738602 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:52.738607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:52.738610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:52.738745 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:52.739410 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:52.739416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:52.739419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:52.739734 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:52.739850 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:52.741746 (+ 1896us) server_negotiation.cc:299] Negotiation successful
0620 15:00:52.742034 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:00:52.744026 10157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:52.715997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43579 (local address 127.2.70.2:44807)
0620 15:00:52.716886 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:52.716937 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:52.719169 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:52.720127 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:52.720141 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:52.721184 (+ 1043us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:52.722344 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:52.722374 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.723936 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.723942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:52.726992 (+ 3050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:52.727014 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.728173 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.728185 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:52.728583 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:52.729615 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:52.729658 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:52.733212 (+ 3554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:52.737618 (+ 4406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:52.737631 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:52.737655 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:52.738448 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:52.738875 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:52.738884 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:52.738890 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:52.739276 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:52.739879 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:52.739895 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:52.740570 (+ 675us) client_negotiation.cc:769] Sending connection context
0620 15:00:52.740980 (+ 410us) client_negotiation.cc:240] Negotiation successful
0620 15:00:52.741837 (+ 857us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":373,"threads_started":1}
I20260620 15:00:52.765440 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:52.751553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:45965 (local address 127.2.70.62:43579)
0620 15:00:52.751740 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:52.751745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:52.752406 (+ 661us) server_negotiation.cc:407] Connection header received
0620 15:00:52.752621 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:52.752626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:52.752702 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:52.752806 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:52.753962 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.755039 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.756434 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.756845 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.756991 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:52.759606 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:52.759611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:52.759615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:52.759646 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:52.762535 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:52.763513 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:52.763517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:52.763519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:52.763654 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:52.764067 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:52.764070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:52.764072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:52.764371 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:52.764478 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:52.765059 (+ 581us) server_negotiation.cc:299] Negotiation successful
0620 15:00:52.765215 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:00:52.766134 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:52.751634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43579 (local address 127.2.70.2:45965)
0620 15:00:52.752293 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:52.752312 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:52.752490 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:52.752831 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:52.752836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:52.753360 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:52.753807 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:52.753818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.755178 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.755184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:52.756297 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:52.756309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:52.757033 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:52.757039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:52.757201 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:52.757961 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:52.757991 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:52.759399 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:52.762684 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:52.762692 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:52.762699 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:52.763383 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:52.763761 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:52.763769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:52.763775 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:52.763960 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:52.764485 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:52.764501 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:52.764769 (+ 268us) client_negotiation.cc:769] Sending connection context
0620 15:00:52.764989 (+ 220us) client_negotiation.cc:240] Negotiation successful
0620 15:00:52.765449 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":386,"threads_started":1}
I20260620 15:00:52.773717 10133 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:42207
I20260620 15:00:52.773741 10267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:42207 every 8 connection(s)
I20260620 15:00:52.774906 10133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:52.776746 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 10133
I20260620 15:00:52.776955 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:52.777057 10268 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43579
I20260620 15:00:52.777282 10268 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:52.777558 2328 external_mini_cluster.cc:1428] Setting key ae20fb247c049b047708983bce96730a
I20260620 15:00:52.778108 10268 heartbeater.cc:511] Master 127.2.70.62:43579 requested a full tablet report, sending...
I20260620 15:00:52.779522 9921 ts_manager.cc:194] Registered new tserver with Master: 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207)
I20260620 15:00:52.780316 9921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:44807
WARNING: no policy specified for oryx/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab.
Entry for principal oryx/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab.
I20260620 15:00:52.824481 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43579
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab
--principal=oryx/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:53.153661 10272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:53.153964 10272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:53.154135 10272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:53.161288 10272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:53.161465 10272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:53.161644 10272 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 15:00:53.161784 10272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:00:53.171885 10272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab
--principal=oryx/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.70.62:43579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:53.173151 10272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:53.174607 10272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:53.187616 10277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:53.187582 10278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:53.187662 10280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:53.188482 10272 server_base.cc:1034] running on GCE node
I20260620 15:00:53.189071 10272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:53.190054 10272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:53.191272 10272 hybrid_clock.cc:648] HybridClock initialized: now 1781967653191092 us; error 154 us; skew 500 ppm
I20260620 15:00:53.191702 10272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:53 dist-test-slave-n5vv krb5kdc[9877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967653, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:53.194947 10272 init.cc:377] Logged in from keytab as oryx/127.2.70.3@KRBTEST.COM (short username oryx)
I20260620 15:00:53.196405 10272 webserver.cc:460] Webserver started at http://127.2.70.3:45133/ using document root <none> and password file <none>
I20260620 15:00:53.197057 10272 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:53.197140 10272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:53.197402 10272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:53.200280 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "023a85e9fb7d405d836e8feb8a6b9aa5"
format_stamp: "Formatted at 2026-06-20 15:00:53 on dist-test-slave-n5vv"
server_key: "658bd3513f489c1c4d6143e73d1b3347"
server_key_iv: "63b9b181553d5de934adde1e727e47da"
server_key_version: "encryptionkey@0"
I20260620 15:00:53.201022 10272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "023a85e9fb7d405d836e8feb8a6b9aa5"
format_stamp: "Formatted at 2026-06-20 15:00:53 on dist-test-slave-n5vv"
server_key: "658bd3513f489c1c4d6143e73d1b3347"
server_key_iv: "63b9b181553d5de934adde1e727e47da"
server_key_version: "encryptionkey@0"
I20260620 15:00:53.206178 10272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260620 15:00:53.209877 10287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:53.211328 10272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:00:53.211525 10272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "023a85e9fb7d405d836e8feb8a6b9aa5"
format_stamp: "Formatted at 2026-06-20 15:00:53 on dist-test-slave-n5vv"
server_key: "658bd3513f489c1c4d6143e73d1b3347"
server_key_iv: "63b9b181553d5de934adde1e727e47da"
server_key_version: "encryptionkey@0"
I20260620 15:00:53.211768 10272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:53.235459 10272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:53.240295 10272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:53.240588 10272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:53.243474 10272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:53.243593 10272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:53.243762 10272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:53.243853 10272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:00:53 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967653, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:53.263451 10129 heartbeater.cc:503] Master 127.2.70.62:43579 was elected leader, sending a full tablet report...
I20260620 15:00:53.271759 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.244011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:34423 (local address 127.2.70.62:43579)
0620 15:00:53.244273 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.244282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.245890 (+ 1608us) server_negotiation.cc:407] Connection header received
0620 15:00:53.248181 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.248191 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.248296 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.248452 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:53.252048 (+ 3596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.253598 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.256977 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.257412 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.257606 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.264219 (+ 6613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:53.264225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:53.264229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:53.264274 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:53.267990 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:53.268980 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:53.268990 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:53.268996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:53.269178 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:53.269798 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:53.269804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:53.269807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:53.270099 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:53.270248 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.271161 (+ 913us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.271382 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:00:53.272815 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.244707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43579 (local address 127.2.70.3:34423)
0620 15:00:53.245708 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.245766 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.247874 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.248752 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.248765 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.250527 (+ 1762us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:53.251719 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.251761 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.253836 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.253849 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.256770 (+ 2921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.256796 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.257713 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.257724 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.258095 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.259234 (+ 1139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:53.259279 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:53.263897 (+ 4618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:53.268146 (+ 4249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:53.268155 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:53.268178 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:53.268795 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:53.269342 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:53.269349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:53.269356 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:53.269657 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:53.270238 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:53.270247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:53.270771 (+ 524us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.271061 (+ 290us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.271645 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":406,"threads_started":1}
I20260620 15:00:53.292891 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.278605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:42685 (local address 127.2.70.62:43579)
0620 15:00:53.278837 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.278846 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.279333 (+ 487us) server_negotiation.cc:407] Connection header received
0620 15:00:53.279545 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.279554 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.279656 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.279799 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:53.281005 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.282146 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.283489 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.283905 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.284067 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.286100 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:53.286109 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:53.286116 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:53.286157 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:53.289966 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:53.290939 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:53.290945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:53.290951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:53.291095 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:53.291474 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:53.291479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:53.291482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:53.291753 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:53.291923 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.292295 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.292514 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:00:53.292985 10360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.278572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43579 (local address 127.2.70.3:42685)
0620 15:00:53.279206 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.279232 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.279402 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.279806 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.279810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.280305 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:53.280857 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.280869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.282284 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.282293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.283341 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.283352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.284109 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.284115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.284285 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.284986 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:53.285005 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:53.285829 (+ 824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:53.290092 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:53.290098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:53.290102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:53.290810 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:53.291201 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:53.291206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:53.291210 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:53.291353 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:53.291870 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:53.291875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:53.292060 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.292200 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.292513 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":352,"threads_started":1}
I20260620 15:00:53.304026 10272 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:41379
I20260620 15:00:53.304055 10406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:41379 every 8 connection(s)
I20260620 15:00:53.305305 10272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:00:53.307554 10407 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43579
I20260620 15:00:53.307574 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 10272
I20260620 15:00:53.307785 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:00:53.307833 10407 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:53.308215 2328 external_mini_cluster.cc:1428] Setting key 4fa1f97b1562b636674b69cd1731196d
I20260620 15:00:53.308568 10407 heartbeater.cc:511] Master 127.2.70.62:43579 requested a full tablet report, sending...
I20260620 15:00:53.310207 9921 ts_manager.cc:194] Registered new tserver with Master: 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379)
I20260620 15:00:53.311147 9921 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:34423
I20260620 15:00:53.311430 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 15:00:53 dist-test-slave-n5vv krb5kdc[9877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967653, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 15:00:53 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:00:53 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:00:53.359652 10017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:53.349665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55666 (local address 127.2.70.62:43579)
0620 15:00:53.349908 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.349915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.350148 (+ 233us) server_negotiation.cc:407] Connection header received
0620 15:00:53.350293 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.350297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.350378 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.350467 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:53.351747 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.352512 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.353769 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.354164 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.354267 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.359313 (+ 5046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55666: BlockingRecv error: recv got EOF from 127.0.0.1:55666 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 20 15:00:53 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967653, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:53.375381 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.362710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.2.70.62:43579)
0620 15:00:53.362954 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.362961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.363148 (+ 187us) server_negotiation.cc:407] Connection header received
0620 15:00:53.363351 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.363355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.363454 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.363541 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:53.364594 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.365379 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.366623 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.367037 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.367147 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.370546 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:53.370551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:53.370554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:53.370584 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:53.372920 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:53.373569 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:53.373573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:53.373575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:53.373691 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:53.374002 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:53.374005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:53.374007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:53.374322 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:53.374523 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.374713 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.374992 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260620 15:00:53.379156 9921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:53.386323 9921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:53.413080 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.399578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:38957 (local address 127.0.0.1:56224)
0620 15:00:53.400750 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.400788 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.400930 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.402114 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.402126 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.402169 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:53.402806 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.402823 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.405509 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.405518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.408995 (+ 3477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.409020 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.412029 (+ 3009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.412036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.412291 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.412355 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.412494 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.412708 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":11232,"threads_started":1}
I20260620 15:00:53.413061 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.399485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.2.70.1:38957)
0620 15:00:53.400624 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.400708 (+ 84us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.400886 (+ 178us) server_negotiation.cc:407] Connection header received
0620 15:00:53.401134 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.401142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.401531 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.401806 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:53.403071 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.405215 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.409308 (+ 4093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.411488 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.411630 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.411875 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.412569 (+ 694us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.412739 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":764,"threads_started":1}
I20260620 15:00:53.417907 10059 tablet_service.cc:1467] Processing CreateTablet for tablet b7fc449ecd03498facf539b6b69d9d0f (DEFAULT_TABLE table=test-table [id=f8332bda93024c1891917ccf1cc226a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:53.419736 10059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fc449ecd03498facf539b6b69d9d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.419752 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.399476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.2.70.2:42207)
0620 15:00:53.400121 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.400133 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.411330 (+ 11197us) server_negotiation.cc:407] Connection header received
0620 15:00:53.412593 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.412608 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.413025 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.413443 (+ 418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:53.414169 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.415503 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.417812 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.418688 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.418841 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.419072 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.419290 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.419459 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":296,"threads_started":1}
I20260620 15:00:53.419780 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.399977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:42207 (local address 127.0.0.1:33954)
0620 15:00:53.411148 (+ 11171us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.411175 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.412365 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.413704 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.413709 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.413745 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:53.414006 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.414014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.415621 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.415630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.417650 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.417662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.418899 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.418906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.419071 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.419111 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.419236 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.419394 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11028,"thread_start_us":340,"threads_started":1}
I20260620 15:00:53.422600 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.402698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54674 (local address 127.2.70.3:41379)
0620 15:00:53.413953 (+ 11255us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.413963 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.413997 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:00:53.414090 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.414096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.414334 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.414542 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:53.415334 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.417309 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.419966 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.421433 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.421588 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.421825 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.422076 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.422264 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11042,"thread_start_us":10787,"threads_started":1}
I20260620 15:00:53.423568 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.401948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:41379 (local address 127.0.0.1:54674)
0620 15:00:53.411889 (+ 9941us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.411911 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.412075 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.414585 (+ 2510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.414591 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.414617 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:53.415017 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.415030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.417517 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.417523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.419734 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.419748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.421689 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.421697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.421877 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.421910 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.422013 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.422154 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9786,"thread_start_us":10256,"threads_started":1}
I20260620 15:00:53.423554 10198 tablet_service.cc:1467] Processing CreateTablet for tablet b7fc449ecd03498facf539b6b69d9d0f (DEFAULT_TABLE table=test-table [id=f8332bda93024c1891917ccf1cc226a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:53.424857 10198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fc449ecd03498facf539b6b69d9d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.429910 10337 tablet_service.cc:1467] Processing CreateTablet for tablet b7fc449ecd03498facf539b6b69d9d0f (DEFAULT_TABLE table=test-table [id=f8332bda93024c1891917ccf1cc226a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:53.431607 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fc449ecd03498facf539b6b69d9d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.434769 10431 tablet_bootstrap.cc:492] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Bootstrap starting.
I20260620 15:00:53.438334 10432 tablet_bootstrap.cc:492] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Bootstrap starting.
I20260620 15:00:53.439321 10431 tablet_bootstrap.cc:654] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.440661 10431 log.cc:824] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:53.442013 10432 tablet_bootstrap.cc:654] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.443257 10432 log.cc:824] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:53.443416 10431 tablet_bootstrap.cc:492] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: No bootstrap required, opened a new log
I20260620 15:00:53.443737 10431 ts_tablet_manager.cc:1397] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260620 15:00:53.446245 10435 tablet_bootstrap.cc:492] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Bootstrap starting.
I20260620 15:00:53.446458 10432 tablet_bootstrap.cc:492] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: No bootstrap required, opened a new log
I20260620 15:00:53.446754 10432 ts_tablet_manager.cc:1397] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:00:53.451247 10435 tablet_bootstrap.cc:654] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.452592 10435 log.cc:824] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:53.454735 10432 raft_consensus.cc:348] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.454633 10431 raft_consensus.cc:348] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.455098 10432 raft_consensus.cc:374] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.455096 10431 raft_consensus.cc:374] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.455196 10432 raft_consensus.cc:729] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abccd676a624f7aa1dd25b15a1ab6c1, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.455191 10431 raft_consensus.cc:729] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378410d6e2894f46848b87c1c20bd3b2, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.455824 10432 consensus_queue.cc:260] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.455940 10431 consensus_queue.cc:260] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.456182 10435 tablet_bootstrap.cc:492] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: No bootstrap required, opened a new log
I20260620 15:00:53.456477 10435 ts_tablet_manager.cc:1397] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260620 15:00:53.457986 10431 ts_tablet_manager.cc:1428] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 15:00:53.458483 10268 heartbeater.cc:503] Master 127.2.70.62:43579 was elected leader, sending a full tablet report...
I20260620 15:00:53.460799 10432 ts_tablet_manager.cc:1428] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20260620 15:00:53.466537 10435 raft_consensus.cc:348] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.466940 10435 raft_consensus.cc:374] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.467049 10435 raft_consensus.cc:729] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023a85e9fb7d405d836e8feb8a6b9aa5, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.467653 10435 consensus_queue.cc:260] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.468837 10407 heartbeater.cc:503] Master 127.2.70.62:43579 was elected leader, sending a full tablet report...
I20260620 15:00:53.469535 10435 ts_tablet_manager.cc:1428] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260620 15:00:53.486096 10439 raft_consensus.cc:482] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:53.486390 10439 raft_consensus.cc:504] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.488135 10439 leader_election.cc:288] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957), 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207)
I20260620 15:00:53.495343 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.488456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:38327 (local address 127.2.70.1:38957)
0620 15:00:53.488673 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.488682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.488960 (+ 278us) server_negotiation.cc:407] Connection header received
0620 15:00:53.489043 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.489049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.489158 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.489256 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:53.489846 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.490825 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.492994 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.494425 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.494538 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.494623 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.495013 (+ 390us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.495142 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:00:53.495458 10296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.488368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:38957 (local address 127.2.70.3:38327)
0620 15:00:53.488662 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.488683 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.488827 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.489288 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.489291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.489317 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:53.489664 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.489676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.490944 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.490950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.492782 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.492794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.494594 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.494600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.494813 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.494852 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.494955 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.495125 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":106,"spinlock_wait_cycles":25472}
I20260620 15:00:53.496490 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.488405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:42207 (local address 127.2.70.3:33023)
0620 15:00:53.489271 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.489294 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.489444 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.489846 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.489850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.489875 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:53.490256 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.490269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.491749 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.491753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.494160 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.494179 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.495828 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.495834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.496010 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.496050 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.496151 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.496255 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":525,"threads_started":1}
I20260620 15:00:53.496642 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.488662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:33023 (local address 127.2.70.2:42207)
0620 15:00:53.488897 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.488906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.489470 (+ 564us) server_negotiation.cc:407] Connection header received
0620 15:00:53.489636 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.489642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.489725 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.489816 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:53.490442 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.491588 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.494349 (+ 2761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.495595 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.495728 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.495826 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.496192 (+ 366us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.496332 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:00:53.496604 10079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fc449ecd03498facf539b6b69d9d0f" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378410d6e2894f46848b87c1c20bd3b2" is_pre_election: true
I20260620 15:00:53.497234 10079 raft_consensus.cc:2454] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 0.
I20260620 15:00:53.497815 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fc449ecd03498facf539b6b69d9d0f" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" is_pre_election: true
I20260620 15:00:53.497898 10290 leader_election.cc:302] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023a85e9fb7d405d836e8feb8a6b9aa5, 378410d6e2894f46848b87c1c20bd3b2; no voters:
I20260620 15:00:53.498291 10218 raft_consensus.cc:2454] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 0.
I20260620 15:00:53.498309 10439 raft_consensus.cc:2781] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:53.498436 10439 raft_consensus.cc:482] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:53.498602 10439 raft_consensus.cc:3037] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.501113 10439 raft_consensus.cc:504] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.501858 10439 leader_election.cc:288] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 election: Requested vote from peers 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957), 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207)
I20260620 15:00:53.502326 10079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fc449ecd03498facf539b6b69d9d0f" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378410d6e2894f46848b87c1c20bd3b2"
I20260620 15:00:53.502424 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7fc449ecd03498facf539b6b69d9d0f" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
I20260620 15:00:53.502534 10079 raft_consensus.cc:3037] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.502691 10218 raft_consensus.cc:3037] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.504875 10079 raft_consensus.cc:2454] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 1.
I20260620 15:00:53.504909 10218 raft_consensus.cc:2454] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 1.
I20260620 15:00:53.505345 10289 leader_election.cc:302] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023a85e9fb7d405d836e8feb8a6b9aa5, 5abccd676a624f7aa1dd25b15a1ab6c1; no voters:
I20260620 15:00:53.505782 10439 raft_consensus.cc:2781] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:53.506279 10439 raft_consensus.cc:686] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 023a85e9fb7d405d836e8feb8a6b9aa5, State: Running, Role: LEADER
I20260620 15:00:53.506919 10439 consensus_queue.cc:237] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
W20260620 15:00:53.508275 10130 tablet.cc:2369] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:53.511817 9920 catalog_manager.cc:5495] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3). New cstate: current_term: 1 leader_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } health_report { overall_health: HEALTHY } } }
W20260620 15:00:53.526935 10269 tablet.cc:2369] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:53.536958 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.531756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.2.70.3:41379)
0620 15:00:53.531965 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.531970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.531988 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:00:53.532180 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.532183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.532263 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.532389 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:53.532745 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.533434 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.534857 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.535269 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.535363 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.535773 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:53.536341 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:53.536446 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.536563 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.536631 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:00:53.545885 10079 raft_consensus.cc:1261] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Refusing update from remote peer 023a85e9fb7d405d836e8feb8a6b9aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:53.545995 10218 raft_consensus.cc:1261] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Refusing update from remote peer 023a85e9fb7d405d836e8feb8a6b9aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:53.547093 10441 consensus_queue.cc:1035] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:53.547551 10439 consensus_queue.cc:1035] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:00:53.557397 10408 tablet.cc:2369] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:00:53.561460 10446 mvcc.cc:205] Tried to move back new op lower bound from 7298939508912222208 to 7298939508768092160. Current Snapshot: MvccSnapshot[applied={T|T < 7298939508912222208}]
I20260620 15:00:53.575533 9921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:53.575868 9921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:53.579663 9921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: Sending DeleteTablet for 3 replicas of tablet b7fc449ecd03498facf539b6b69d9d0f
I20260620 15:00:53.581072 10059 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fc449ecd03498facf539b6b69d9d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:53 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:56224
I20260620 15:00:53.581111 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fc449ecd03498facf539b6b69d9d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:53 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:54674
I20260620 15:00:53.582105 10454 tablet_replica.cc:330] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: stopping tablet replica
I20260620 15:00:53.582022 10198 tablet_service.cc:1514] Processing DeleteTablet for tablet b7fc449ecd03498facf539b6b69d9d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:53 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:33954
I20260620 15:00:53.582362 9920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 15:00:53.582600 10454 raft_consensus.cc:2229] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 15:00:53.583081 9920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:53.583283 10454 raft_consensus.cc:2258] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:53.592096 10058 tablet_service.cc:1467] Processing CreateTablet for tablet 264dbd6288d041a594ee3729d52cdbaf (DEFAULT_TABLE table=test-table [id=d0ccccf3ad9d4007a93e1ee8d6803861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:53.592597 10058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264dbd6288d041a594ee3729d52cdbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.598539 10336 tablet_service.cc:1467] Processing CreateTablet for tablet 264dbd6288d041a594ee3729d52cdbaf (DEFAULT_TABLE table=test-table [id=d0ccccf3ad9d4007a93e1ee8d6803861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:53.599134 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264dbd6288d041a594ee3729d52cdbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.600332 10454 ts_tablet_manager.cc:1905] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:53.602286 10197 tablet_service.cc:1467] Processing CreateTablet for tablet 264dbd6288d041a594ee3729d52cdbaf (DEFAULT_TABLE table=test-table [id=d0ccccf3ad9d4007a93e1ee8d6803861]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:53.602905 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 264dbd6288d041a594ee3729d52cdbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.607275 10454 ts_tablet_manager.cc:1918] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:53.607545 10455 tablet_replica.cc:330] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: stopping tablet replica
I20260620 15:00:53.608031 10455 raft_consensus.cc:2229] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:53.608549 10455 raft_consensus.cc:2258] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:53.609330 10456 tablet_replica.cc:330] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: stopping tablet replica
I20260620 15:00:53.609790 10456 raft_consensus.cc:2229] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:53.610484 10456 raft_consensus.cc:2258] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:53.612884 10456 ts_tablet_manager.cc:1905] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:53.613154 10454 log.cc:1196] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/b7fc449ecd03498facf539b6b69d9d0f
I20260620 15:00:53.613821 10454 ts_tablet_manager.cc:1939] T b7fc449ecd03498facf539b6b69d9d0f P 378410d6e2894f46848b87c1c20bd3b2: Deleting consensus metadata
I20260620 15:00:53.615811 9908 catalog_manager.cc:4841] TS 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957): tablet b7fc449ecd03498facf539b6b69d9d0f (table test-table [id=f8332bda93024c1891917ccf1cc226a2]) successfully deleted
I20260620 15:00:53.615903 10455 ts_tablet_manager.cc:1905] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:53.621465 10431 tablet_bootstrap.cc:492] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: Bootstrap starting.
I20260620 15:00:53.622786 10431 tablet_bootstrap.cc:654] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.622943 10455 ts_tablet_manager.cc:1918] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:53.623111 10455 log.cc:1196] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/b7fc449ecd03498facf539b6b69d9d0f
I20260620 15:00:53.623646 10455 ts_tablet_manager.cc:1939] T b7fc449ecd03498facf539b6b69d9d0f P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting consensus metadata
I20260620 15:00:53.625447 9907 catalog_manager.cc:4841] TS 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207): tablet b7fc449ecd03498facf539b6b69d9d0f (table test-table [id=f8332bda93024c1891917ccf1cc226a2]) successfully deleted
I20260620 15:00:53.625595 10431 tablet_bootstrap.cc:492] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: No bootstrap required, opened a new log
I20260620 15:00:53.625725 10431 ts_tablet_manager.cc:1397] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260620 15:00:53.626336 10431 raft_consensus.cc:348] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.626535 10431 raft_consensus.cc:374] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.626639 10431 raft_consensus.cc:729] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378410d6e2894f46848b87c1c20bd3b2, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.626924 10431 consensus_queue.cc:260] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.627429 10431 ts_tablet_manager.cc:1428] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 15:00:53.629567 10435 tablet_bootstrap.cc:492] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: Bootstrap starting.
I20260620 15:00:53.630240 10432 tablet_bootstrap.cc:492] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: Bootstrap starting.
I20260620 15:00:53.631353 10435 tablet_bootstrap.cc:654] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.631500 10432 tablet_bootstrap.cc:654] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.632992 10456 ts_tablet_manager.cc:1918] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:53.633198 10456 log.cc:1196] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/b7fc449ecd03498facf539b6b69d9d0f
I20260620 15:00:53.633845 10456 ts_tablet_manager.cc:1939] T b7fc449ecd03498facf539b6b69d9d0f P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting consensus metadata
I20260620 15:00:53.634837 10432 tablet_bootstrap.cc:492] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: No bootstrap required, opened a new log
I20260620 15:00:53.635004 10432 ts_tablet_manager.cc:1397] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 15:00:53.635684 10432 raft_consensus.cc:348] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.635874 9907 catalog_manager.cc:4841] TS 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379): tablet b7fc449ecd03498facf539b6b69d9d0f (table test-table [id=f8332bda93024c1891917ccf1cc226a2]) successfully deleted
I20260620 15:00:53.635913 10432 raft_consensus.cc:374] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.636020 10432 raft_consensus.cc:729] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abccd676a624f7aa1dd25b15a1ab6c1, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.636307 10432 consensus_queue.cc:260] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.636859 10432 ts_tablet_manager.cc:1428] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260620 15:00:53.637157 10435 tablet_bootstrap.cc:492] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: No bootstrap required, opened a new log
I20260620 15:00:53.637300 10435 ts_tablet_manager.cc:1397] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20260620 15:00:53.638178 10435 raft_consensus.cc:348] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.638474 10435 raft_consensus.cc:374] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.638569 10435 raft_consensus.cc:729] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023a85e9fb7d405d836e8feb8a6b9aa5, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.638839 10435 consensus_queue.cc:260] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.639629 10435 ts_tablet_manager.cc:1428] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 15:00:53.776343 10449 raft_consensus.cc:482] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:53.776605 10449 raft_consensus.cc:504] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.777417 10449 leader_election.cc:288] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957), 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207)
I20260620 15:00:53.778041 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264dbd6288d041a594ee3729d52cdbaf" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" is_pre_election: true
I20260620 15:00:53.778194 10079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264dbd6288d041a594ee3729d52cdbaf" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378410d6e2894f46848b87c1c20bd3b2" is_pre_election: true
I20260620 15:00:53.778277 10218 raft_consensus.cc:2454] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 0.
I20260620 15:00:53.778425 10079 raft_consensus.cc:2454] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 0.
I20260620 15:00:53.778927 10289 leader_election.cc:302] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023a85e9fb7d405d836e8feb8a6b9aa5, 5abccd676a624f7aa1dd25b15a1ab6c1; no voters:
I20260620 15:00:53.779237 10449 raft_consensus.cc:2781] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:53.779358 10449 raft_consensus.cc:482] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:53.779439 10449 raft_consensus.cc:3037] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.782634 10449 raft_consensus.cc:504] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.783617 10079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264dbd6288d041a594ee3729d52cdbaf" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378410d6e2894f46848b87c1c20bd3b2"
I20260620 15:00:53.783811 10079 raft_consensus.cc:3037] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.783852 10449 leader_election.cc:288] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 election: Requested vote from peers 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957), 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207)
I20260620 15:00:53.784166 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "264dbd6288d041a594ee3729d52cdbaf" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
I20260620 15:00:53.784365 10218 raft_consensus.cc:3037] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.786758 10079 raft_consensus.cc:2454] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 1.
I20260620 15:00:53.787173 10290 leader_election.cc:302] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023a85e9fb7d405d836e8feb8a6b9aa5, 378410d6e2894f46848b87c1c20bd3b2; no voters:
I20260620 15:00:53.787303 10218 raft_consensus.cc:2454] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 1.
I20260620 15:00:53.787501 10449 raft_consensus.cc:2781] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:53.787633 10449 raft_consensus.cc:686] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 023a85e9fb7d405d836e8feb8a6b9aa5, State: Running, Role: LEADER
I20260620 15:00:53.787920 10449 consensus_queue.cc:237] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.790804 9920 catalog_manager.cc:5495] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3). New cstate: current_term: 1 leader_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } health_report { overall_health: HEALTHY } } }
Jun 20 15:00:53 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967651, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.62@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:53.850335 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.836044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43579 (local address 127.0.0.1:55684)
0620 15:00:53.836617 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.836632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.836804 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.837125 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.837129 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.837950 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:53.838227 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.838238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.839466 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.839474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.840525 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.840537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.841247 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.841253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.841432 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.842234 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:53.842251 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:53.844380 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:53.847721 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:53.847733 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:53.847741 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:53.848133 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:53.848495 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:53.848500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:53.848504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:53.848648 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:53.849197 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:53.849204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:53.849364 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.849557 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.849856 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":325,"threads_started":1}
I20260620 15:00:53.850358 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.835988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55684 (local address 127.2.70.62:43579)
0620 15:00:53.836200 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.836204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.836726 (+ 522us) server_negotiation.cc:407] Connection header received
0620 15:00:53.836927 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.836932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.836999 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.837107 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:53.838432 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.839333 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.840674 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.841063 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.841208 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.844586 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:53.844591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:53.844594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:53.844621 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:53.847584 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:53.848259 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:53.848267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:53.848271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:53.848382 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:53.848771 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:53.848776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:53.848780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:53.849063 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:53.849221 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.849656 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.849974 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":98}
I20260620 15:00:53.856585 9920 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:55684:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 15:00:53.858000 9920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:53.865231 10058 tablet_service.cc:1467] Processing CreateTablet for tablet af43cf9ac83148a49318837d10ec2854 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8fb2a5cdae8f4e5dbf1e4dbef0f86133]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:53.865430 10197 tablet_service.cc:1467] Processing CreateTablet for tablet af43cf9ac83148a49318837d10ec2854 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8fb2a5cdae8f4e5dbf1e4dbef0f86133]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:53.865844 10058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af43cf9ac83148a49318837d10ec2854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.865811 10336 tablet_service.cc:1467] Processing CreateTablet for tablet af43cf9ac83148a49318837d10ec2854 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8fb2a5cdae8f4e5dbf1e4dbef0f86133]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 15:00:53.865993 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af43cf9ac83148a49318837d10ec2854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.866308 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af43cf9ac83148a49318837d10ec2854. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:53.873206 10431 tablet_bootstrap.cc:492] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: Bootstrap starting.
I20260620 15:00:53.873459 10432 tablet_bootstrap.cc:492] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: Bootstrap starting.
I20260620 15:00:53.874987 10431 tablet_bootstrap.cc:654] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.875154 10432 tablet_bootstrap.cc:654] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.876037 10435 tablet_bootstrap.cc:492] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: Bootstrap starting.
I20260620 15:00:53.877746 10435 tablet_bootstrap.cc:654] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:53.880400 10431 tablet_bootstrap.cc:492] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: No bootstrap required, opened a new log
I20260620 15:00:53.880901 10431 ts_tablet_manager.cc:1397] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260620 15:00:53.882014 10431 raft_consensus.cc:348] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.882391 10431 raft_consensus.cc:374] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.882519 10431 raft_consensus.cc:729] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378410d6e2894f46848b87c1c20bd3b2, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.883044 10431 consensus_queue.cc:260] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.883244 10435 tablet_bootstrap.cc:492] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: No bootstrap required, opened a new log
I20260620 15:00:53.883419 10435 ts_tablet_manager.cc:1397] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260620 15:00:53.883558 10438 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.883898 10438 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:53.884203 10435 raft_consensus.cc:348] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.884460 10435 raft_consensus.cc:374] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.884543 10435 raft_consensus.cc:729] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023a85e9fb7d405d836e8feb8a6b9aa5, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.884896 10431 ts_tablet_manager.cc:1428] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260620 15:00:53.885005 10435 consensus_queue.cc:260] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.885452 10449 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.885774 10449 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:53.886417 10435 ts_tablet_manager.cc:1428] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 15:00:53.887699 10432 tablet_bootstrap.cc:492] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: No bootstrap required, opened a new log
I20260620 15:00:53.887869 10432 ts_tablet_manager.cc:1397] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.008s
I20260620 15:00:53.888583 10432 raft_consensus.cc:348] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.888823 10432 raft_consensus.cc:374] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:53.888911 10432 raft_consensus.cc:729] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abccd676a624f7aa1dd25b15a1ab6c1, State: Initialized, Role: FOLLOWER
I20260620 15:00:53.889197 10432 consensus_queue.cc:260] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.889678 10437 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.890048 10437 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:53.890632 10432 ts_tablet_manager.cc:1428] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 15:00:53.913194 10438 raft_consensus.cc:482] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:53.913444 10438 raft_consensus.cc:504] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.915138 10438 leader_election.cc:288] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207), 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379)
I20260620 15:00:53.923683 10425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.915746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:53123 (local address 127.2.70.2:42207)
0620 15:00:53.915953 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.915961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.916245 (+ 284us) server_negotiation.cc:407] Connection header received
0620 15:00:53.916581 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.916590 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.916691 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.916793 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:53.917696 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.918854 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.921502 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.922619 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.922754 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.922861 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.923242 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.923369 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:00:53.924014 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.915432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:42207 (local address 127.2.70.1:53123)
0620 15:00:53.916148 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.916170 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.916373 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.917041 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.917044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.917081 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:53.917489 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.917520 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.919291 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.919297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.921308 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.921320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.922850 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.922857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.923042 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.923087 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.923193 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.923333 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":54,"thread_start_us":431,"threads_started":1}
I20260620 15:00:53.924353 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.916428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:32847 (local address 127.2.70.3:41379)
0620 15:00:53.916611 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.916617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.917247 (+ 630us) server_negotiation.cc:407] Connection header received
0620 15:00:53.917442 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.917448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.917557 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.917652 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:00:53.918179 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.919020 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.921912 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.923137 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.923286 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.923375 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.923919 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.924027 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:00:53.924764 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af43cf9ac83148a49318837d10ec2854" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" is_pre_election: true
I20260620 15:00:53.924911 10471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.916280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:41379 (local address 127.2.70.1:32847)
0620 15:00:53.917103 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.917124 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.917293 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.917711 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.917716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.917738 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:00:53.918024 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.918031 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.919160 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.919166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.921734 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.921748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.923535 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.923542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.923722 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.923765 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.924543 (+ 778us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.924662 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":1,"thread_start_us":490,"threads_started":1}
I20260620 15:00:53.925052 10218 raft_consensus.cc:2454] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 0.
I20260620 15:00:53.925637 10010 leader_election.cc:302] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378410d6e2894f46848b87c1c20bd3b2, 5abccd676a624f7aa1dd25b15a1ab6c1; no voters:
I20260620 15:00:53.926127 10438 raft_consensus.cc:2781] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:53.926249 10438 raft_consensus.cc:482] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:53.926344 10438 raft_consensus.cc:3037] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.926092 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af43cf9ac83148a49318837d10ec2854" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" is_pre_election: true
I20260620 15:00:53.926613 10362 raft_consensus.cc:2454] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 0.
I20260620 15:00:53.928877 10438 raft_consensus.cc:504] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.929533 10438 leader_election.cc:288] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207), 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379)
I20260620 15:00:53.930055 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af43cf9ac83148a49318837d10ec2854" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
I20260620 15:00:53.930297 10218 raft_consensus.cc:3037] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.930263 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af43cf9ac83148a49318837d10ec2854" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5"
I20260620 15:00:53.930476 10362 raft_consensus.cc:3037] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:53.932948 10362 raft_consensus.cc:2454] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 1.
I20260620 15:00:53.933219 10218 raft_consensus.cc:2454] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 1.
I20260620 15:00:53.933305 10010 leader_election.cc:302] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 023a85e9fb7d405d836e8feb8a6b9aa5, 378410d6e2894f46848b87c1c20bd3b2; no voters:
I20260620 15:00:53.933779 10438 raft_consensus.cc:2781] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:53.934284 10438 raft_consensus.cc:686] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 378410d6e2894f46848b87c1c20bd3b2, State: Running, Role: LEADER
I20260620 15:00:53.934875 10438 consensus_queue.cc:237] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:53.936760 10472 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: TxnStatusTablet state changed. Reason: New leader 378410d6e2894f46848b87c1c20bd3b2. Latest consensus state: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.937170 10472 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:53.937902 10474 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 15:00:53.938131 10474 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 15:00:53.938736 9920 catalog_manager.cc:5495] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:53.960095 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.951947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:38957 (local address 127.0.0.1:56234)
0620 15:00:53.952284 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:53.952305 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:53.952439 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:53.953289 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:53.953294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:53.953307 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:53.953530 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.953543 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.955067 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.955073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.956721 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:53.956738 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.957856 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.957862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:53.958043 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.958112 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:53.959406 (+ 1294us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:53.959430 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:00:53.959525 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:00:53.959686 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260620 15:00:53.960886 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:53.952049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.2.70.1:38957)
0620 15:00:53.952741 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:53.952749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:53.952771 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:00:53.952855 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:53.952860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:53.952962 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:53.953095 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:53.953707 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.954901 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.956893 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:53.957584 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:53.957724 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:53.958347 (+ 623us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:53.959194 (+ 847us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:53.959345 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:53.960440 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0620 15:00:53.960566 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564}
I20260620 15:00:53.965389 10218 raft_consensus.cc:1261] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Refusing update from remote peer 378410d6e2894f46848b87c1c20bd3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:53.965566 10362 raft_consensus.cc:1261] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Refusing update from remote peer 378410d6e2894f46848b87c1c20bd3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:53.966254 10438 consensus_queue.cc:1035] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:53.966838 10472 consensus_queue.cc:1035] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:53.979779 10449 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: TxnStatusTablet state changed. Reason: New leader 378410d6e2894f46848b87c1c20bd3b2. Latest consensus state: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.980137 10449 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:53.984015 10472 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.984267 10472 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:53.984607 10445 mvcc.cc:205] Tried to move back new op lower bound from 7298939510634491904 to 7298939510520549376. Current Snapshot: MvccSnapshot[applied={T|T < 7298939510634491904 or (T in {7298939510634491904})}]
I20260620 15:00:53.987115 10444 mvcc.cc:205] Tried to move back new op lower bound from 7298939510634491904 to 7298939510520549376. Current Snapshot: MvccSnapshot[applied={T|T < 7298939510634491904}]
I20260620 15:00:53.987394 10449 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.987653 10449 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 023a85e9fb7d405d836e8feb8a6b9aa5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:53.993907 10437 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: TxnStatusTablet state changed. Reason: New leader 378410d6e2894f46848b87c1c20bd3b2. Latest consensus state: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.994220 10437 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:53.994217 10438 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.994541 10438 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 378410d6e2894f46848b87c1c20bd3b2: This TxnStatusTablet replica's current role is: LEADER
I20260620 15:00:53.997428 10437 tablet_replica.cc:439] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } } }
I20260620 15:00:53.997648 10437 tablet_replica.cc:442] T af43cf9ac83148a49318837d10ec2854 P 5abccd676a624f7aa1dd25b15a1ab6c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 15:00:54.018260 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.012029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:51205 (local address 127.2.70.1:38957)
0620 15:00:54.012199 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.012207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.012960 (+ 753us) server_negotiation.cc:407] Connection header received
0620 15:00:54.013230 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.013246 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.013312 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.013418 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:54.013872 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.014688 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.016365 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.016900 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.017035 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.017467 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:54.017633 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:54.017734 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:54.017871 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 15:00:54.017946 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:00:54.018608 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.012158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:38957 (local address 127.2.70.3:51205)
0620 15:00:54.012855 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:54.012872 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:54.013065 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:54.013439 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:54.013444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:54.013460 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:54.013723 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:54.013733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.014818 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.014830 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:54.016173 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:54.016188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.017069 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.017077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:54.017218 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.017280 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:54.017744 (+ 464us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:54.017758 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 15:00:54.017850 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:00:54.017992 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":363,"threads_started":1}
I20260620 15:00:54.029042 10079 raft_consensus.cc:1261] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Refusing update from remote peer 023a85e9fb7d405d836e8feb8a6b9aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:54.029708 10439 consensus_queue.cc:1035] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:54.030143 10218 raft_consensus.cc:1261] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Refusing update from remote peer 023a85e9fb7d405d836e8feb8a6b9aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:54.030743 10439 consensus_queue.cc:1035] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:54.077138 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.067535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:36233 (local address 127.2.70.3:41379)
0620 15:00:54.067717 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.067723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.070804 (+ 3081us) server_negotiation.cc:407] Connection header received
0620 15:00:54.071126 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.071132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.071249 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.071318 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:54.072156 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.073036 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.074784 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.075436 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.075547 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.076276 (+ 729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:54.076408 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:54.076475 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:54.076706 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 15:00:54.076792 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:00:54.078151 10485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.067081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:41379 (local address 127.2.70.1:36233)
0620 15:00:54.070702 (+ 3621us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:54.070726 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:54.070941 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:54.071517 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:54.071522 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:54.071534 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 15:00:54.071973 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:54.071987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.073176 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.073179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:54.074621 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:54.074635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.075737 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.075745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:54.076003 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.076080 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 15:00:54.076560 (+ 480us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 15:00:54.076570 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:00:54.077694 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0620 15:00:54.077841 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3205,"thread_start_us":2734,"threads_started":1}
I20260620 15:00:54.117421 9921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:54.117744 9921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:54.120919 9921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: Sending DeleteTablet for 3 replicas of tablet 264dbd6288d041a594ee3729d52cdbaf
I20260620 15:00:54.121748 10058 tablet_service.cc:1514] Processing DeleteTablet for tablet 264dbd6288d041a594ee3729d52cdbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:56224
I20260620 15:00:54.122184 10454 tablet_replica.cc:330] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: stopping tablet replica
I20260620 15:00:54.122200 10197 tablet_service.cc:1514] Processing DeleteTablet for tablet 264dbd6288d041a594ee3729d52cdbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:33954
I20260620 15:00:54.122402 10454 raft_consensus.cc:2229] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:54.122475 10455 tablet_replica.cc:330] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: stopping tablet replica
I20260620 15:00:54.122620 10454 raft_consensus.cc:2258] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.122654 10455 raft_consensus.cc:2229] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:54.122879 10455 raft_consensus.cc:2258] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.123127 10336 tablet_service.cc:1514] Processing DeleteTablet for tablet 264dbd6288d041a594ee3729d52cdbaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:54674
I20260620 15:00:54.123415 10456 tablet_replica.cc:330] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: stopping tablet replica
I20260620 15:00:54.123600 10456 raft_consensus.cc:2229] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:54.123914 10456 raft_consensus.cc:2258] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.124333 10454 ts_tablet_manager.cc:1905] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.124532 10455 ts_tablet_manager.cc:1905] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.126415 10456 ts_tablet_manager.cc:1905] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.132027 10454 ts_tablet_manager.cc:1918] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:54.132196 10454 log.cc:1196] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/264dbd6288d041a594ee3729d52cdbaf
I20260620 15:00:54.132686 10454 ts_tablet_manager.cc:1939] T 264dbd6288d041a594ee3729d52cdbaf P 378410d6e2894f46848b87c1c20bd3b2: Deleting consensus metadata
I20260620 15:00:54.133137 10456 ts_tablet_manager.cc:1918] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:54.133289 10456 log.cc:1196] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/264dbd6288d041a594ee3729d52cdbaf
I20260620 15:00:54.133708 10456 ts_tablet_manager.cc:1939] T 264dbd6288d041a594ee3729d52cdbaf P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting consensus metadata
I20260620 15:00:54.133826 9908 catalog_manager.cc:4841] TS 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957): tablet 264dbd6288d041a594ee3729d52cdbaf (table test-table [id=d0ccccf3ad9d4007a93e1ee8d6803861]) successfully deleted
I20260620 15:00:54.134768 9907 catalog_manager.cc:4841] TS 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379): tablet 264dbd6288d041a594ee3729d52cdbaf (table test-table [id=d0ccccf3ad9d4007a93e1ee8d6803861]) successfully deleted
I20260620 15:00:54.136193 10455 ts_tablet_manager.cc:1918] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:54.136349 10455 log.cc:1196] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/264dbd6288d041a594ee3729d52cdbaf
I20260620 15:00:54.136775 10455 ts_tablet_manager.cc:1939] T 264dbd6288d041a594ee3729d52cdbaf P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting consensus metadata
I20260620 15:00:54.137838 9907 catalog_manager.cc:4841] TS 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207): tablet 264dbd6288d041a594ee3729d52cdbaf (table test-table [id=d0ccccf3ad9d4007a93e1ee8d6803861]) successfully deleted
Jun 20 15:00:54 dist-test-slave-n5vv krb5kdc[9877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 15:00:54 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:00:54 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:00:54.187861 10017 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:00:54.172719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55690 (local address 127.2.70.62:43579)
0620 15:00:54.172943 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.172950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.173326 (+ 376us) server_negotiation.cc:407] Connection header received
0620 15:00:54.173468 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.173474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.173576 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.173695 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:54.175249 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.176391 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.177783 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.178492 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.178624 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.187617 (+ 8993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55690: BlockingRecv error: recv got EOF from 127.0.0.1:55690 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 15:00:54 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM
I20260620 15:00:54.205983 10017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.190805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55696 (local address 127.2.70.62:43579)
0620 15:00:54.191007 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.191015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.191410 (+ 395us) server_negotiation.cc:407] Connection header received
0620 15:00:54.191576 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.191582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.191659 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.191771 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:54.192995 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.194159 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.195425 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.196082 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.196207 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.199968 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:54.199975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:54.199980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:54.200020 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:54.203213 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:54.203835 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:54.203840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:54.203843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:54.203995 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:54.204432 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:54.204437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:54.204441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:54.204811 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:54.205066 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:54.205337 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 15:00:54.205588 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 15:00:54.209805 9921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:54.210538 9921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:54.219475 10058 tablet_service.cc:1467] Processing CreateTablet for tablet be038e0802194b8489f93ab10848de4d (DEFAULT_TABLE table=test-table [id=223358e5f0bc4e8eac3167892fef3356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:54.219820 10336 tablet_service.cc:1467] Processing CreateTablet for tablet be038e0802194b8489f93ab10848de4d (DEFAULT_TABLE table=test-table [id=223358e5f0bc4e8eac3167892fef3356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:54.219838 10197 tablet_service.cc:1467] Processing CreateTablet for tablet be038e0802194b8489f93ab10848de4d (DEFAULT_TABLE table=test-table [id=223358e5f0bc4e8eac3167892fef3356]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:54.219983 10058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be038e0802194b8489f93ab10848de4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:54.220224 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be038e0802194b8489f93ab10848de4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:54.220293 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be038e0802194b8489f93ab10848de4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:54.226891 10432 tablet_bootstrap.cc:492] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: Bootstrap starting.
I20260620 15:00:54.227039 10435 tablet_bootstrap.cc:492] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: Bootstrap starting.
I20260620 15:00:54.227044 10431 tablet_bootstrap.cc:492] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: Bootstrap starting.
I20260620 15:00:54.228521 10432 tablet_bootstrap.cc:654] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:54.228659 10435 tablet_bootstrap.cc:654] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:54.228685 10431 tablet_bootstrap.cc:654] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:54.238117 10431 tablet_bootstrap.cc:492] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: No bootstrap required, opened a new log
I20260620 15:00:54.238186 10435 tablet_bootstrap.cc:492] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: No bootstrap required, opened a new log
I20260620 15:00:54.238329 10435 ts_tablet_manager.cc:1397] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20260620 15:00:54.238341 10431 ts_tablet_manager.cc:1397] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 15:00:54.239128 10431 raft_consensus.cc:348] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.239187 10435 raft_consensus.cc:348] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.239418 10431 raft_consensus.cc:374] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:54.239460 10435 raft_consensus.cc:374] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:54.239529 10431 raft_consensus.cc:729] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378410d6e2894f46848b87c1c20bd3b2, State: Initialized, Role: FOLLOWER
I20260620 15:00:54.239555 10435 raft_consensus.cc:729] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023a85e9fb7d405d836e8feb8a6b9aa5, State: Initialized, Role: FOLLOWER
I20260620 15:00:54.239810 10435 consensus_queue.cc:260] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.239979 10431 consensus_queue.cc:260] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.241297 10435 ts_tablet_manager.cc:1428] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 15:00:54.242056 10431 ts_tablet_manager.cc:1428] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 15:00:54.245690 10432 tablet_bootstrap.cc:492] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: No bootstrap required, opened a new log
I20260620 15:00:54.245915 10432 ts_tablet_manager.cc:1397] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.011s
I20260620 15:00:54.246730 10432 raft_consensus.cc:348] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.247025 10432 raft_consensus.cc:374] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:54.247126 10432 raft_consensus.cc:729] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abccd676a624f7aa1dd25b15a1ab6c1, State: Initialized, Role: FOLLOWER
I20260620 15:00:54.247395 10432 consensus_queue.cc:260] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.247962 10432 ts_tablet_manager.cc:1428] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260620 15:00:54.306375 10487 raft_consensus.cc:482] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:54.306619 10487 raft_consensus.cc:504] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.307415 10487 leader_election.cc:288] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207), 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379)
I20260620 15:00:54.308010 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be038e0802194b8489f93ab10848de4d" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" is_pre_election: true
I20260620 15:00:54.307999 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be038e0802194b8489f93ab10848de4d" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" is_pre_election: true
I20260620 15:00:54.308239 10218 raft_consensus.cc:2454] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 0.
I20260620 15:00:54.308235 10362 raft_consensus.cc:2454] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 0.
I20260620 15:00:54.308727 10010 leader_election.cc:302] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378410d6e2894f46848b87c1c20bd3b2, 5abccd676a624f7aa1dd25b15a1ab6c1; no voters:
I20260620 15:00:54.309002 10487 raft_consensus.cc:2781] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:54.309139 10487 raft_consensus.cc:482] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:54.309237 10487 raft_consensus.cc:3037] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:54.311853 10487 raft_consensus.cc:504] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.312515 10487 leader_election.cc:288] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207), 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379)
I20260620 15:00:54.313084 10362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be038e0802194b8489f93ab10848de4d" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5"
I20260620 15:00:54.313087 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be038e0802194b8489f93ab10848de4d" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
I20260620 15:00:54.313320 10362 raft_consensus.cc:3037] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:54.313339 10218 raft_consensus.cc:3037] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:54.316433 10218 raft_consensus.cc:2454] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 1.
I20260620 15:00:54.316846 10362 raft_consensus.cc:2454] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 1.
I20260620 15:00:54.317191 10010 leader_election.cc:302] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378410d6e2894f46848b87c1c20bd3b2, 5abccd676a624f7aa1dd25b15a1ab6c1; no voters:
I20260620 15:00:54.317610 10487 raft_consensus.cc:2781] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:54.317816 10487 raft_consensus.cc:686] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 LEADER]: Becoming Leader. State: Replica: 378410d6e2894f46848b87c1c20bd3b2, State: Running, Role: LEADER
I20260620 15:00:54.318152 10487 consensus_queue.cc:237] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.321303 9921 catalog_manager.cc:5495] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "378410d6e2894f46848b87c1c20bd3b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } health_report { overall_health: UNKNOWN } } }
I20260620 15:00:54.349226 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.343594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56240 (local address 127.2.70.1:38957)
0620 15:00:54.344067 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.344071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.344085 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:00:54.344137 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.344141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.344199 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.344310 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:54.344682 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.345645 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.347440 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.347914 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.348058 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.348484 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:54.348615 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:54.348714 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:54.348853 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 15:00:54.348945 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363}
I20260620 15:00:54.352838 10218 raft_consensus.cc:1261] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Refusing update from remote peer 378410d6e2894f46848b87c1c20bd3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:54.352859 10362 raft_consensus.cc:1261] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Refusing update from remote peer 378410d6e2894f46848b87c1c20bd3b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:54.353510 10438 consensus_queue.cc:1035] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:54.353803 10472 consensus_queue.cc:1035] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:54.370165 9921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:54.370383 9921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:54.373199 9921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: Sending DeleteTablet for 3 replicas of tablet be038e0802194b8489f93ab10848de4d
I20260620 15:00:54.373962 10058 tablet_service.cc:1514] Processing DeleteTablet for tablet be038e0802194b8489f93ab10848de4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:56224
I20260620 15:00:54.374164 10336 tablet_service.cc:1514] Processing DeleteTablet for tablet be038e0802194b8489f93ab10848de4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:54674
I20260620 15:00:54.374269 10454 tablet_replica.cc:330] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: stopping tablet replica
I20260620 15:00:54.374253 10197 tablet_service.cc:1514] Processing DeleteTablet for tablet be038e0802194b8489f93ab10848de4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:33954
I20260620 15:00:54.374450 10456 tablet_replica.cc:330] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: stopping tablet replica
I20260620 15:00:54.374491 10454 raft_consensus.cc:2229] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:54.374609 10456 raft_consensus.cc:2229] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:54.374828 10456 raft_consensus.cc:2258] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.375145 10455 tablet_replica.cc:330] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: stopping tablet replica
I20260620 15:00:54.375216 10454 raft_consensus.cc:2258] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.375324 10455 raft_consensus.cc:2229] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:54.375547 10455 raft_consensus.cc:2258] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.375681 9921 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:00:54.376344 9921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:00:54.376708 10454 ts_tablet_manager.cc:1905] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.376745 10456 ts_tablet_manager.cc:1905] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.381790 10456 ts_tablet_manager.cc:1918] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:54.381943 10456 log.cc:1196] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/be038e0802194b8489f93ab10848de4d
I20260620 15:00:54.382301 10456 ts_tablet_manager.cc:1939] T be038e0802194b8489f93ab10848de4d P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting consensus metadata
I20260620 15:00:54.383399 9907 catalog_manager.cc:4841] TS 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379): tablet be038e0802194b8489f93ab10848de4d (table test-table [id=223358e5f0bc4e8eac3167892fef3356]) successfully deleted
I20260620 15:00:54.383601 10454 ts_tablet_manager.cc:1918] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:54.383754 10454 log.cc:1196] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/be038e0802194b8489f93ab10848de4d
I20260620 15:00:54.384195 10454 ts_tablet_manager.cc:1939] T be038e0802194b8489f93ab10848de4d P 378410d6e2894f46848b87c1c20bd3b2: Deleting consensus metadata
I20260620 15:00:54.385353 9908 catalog_manager.cc:4841] TS 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957): tablet be038e0802194b8489f93ab10848de4d (table test-table [id=223358e5f0bc4e8eac3167892fef3356]) successfully deleted
I20260620 15:00:54.386097 10455 ts_tablet_manager.cc:1905] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.390288 10197 tablet_service.cc:1467] Processing CreateTablet for tablet fc59179d0461447085c79c0df767680d (DEFAULT_TABLE table=test-table [id=364866ff56b44f96b5cc7a1ae5e6eef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:54.390290 10058 tablet_service.cc:1467] Processing CreateTablet for tablet fc59179d0461447085c79c0df767680d (DEFAULT_TABLE table=test-table [id=364866ff56b44f96b5cc7a1ae5e6eef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:54.390722 10058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc59179d0461447085c79c0df767680d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:54.390748 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc59179d0461447085c79c0df767680d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:54.393898 10336 tablet_service.cc:1467] Processing CreateTablet for tablet fc59179d0461447085c79c0df767680d (DEFAULT_TABLE table=test-table [id=364866ff56b44f96b5cc7a1ae5e6eef0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:00:54.394230 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc59179d0461447085c79c0df767680d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:54.397732 10431 tablet_bootstrap.cc:492] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: Bootstrap starting.
I20260620 15:00:54.399560 10431 tablet_bootstrap.cc:654] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:54.400888 10455 ts_tablet_manager.cc:1918] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:00:54.401028 10455 log.cc:1196] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/be038e0802194b8489f93ab10848de4d
I20260620 15:00:54.401546 10455 ts_tablet_manager.cc:1939] T be038e0802194b8489f93ab10848de4d P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting consensus metadata
I20260620 15:00:54.401904 10435 tablet_bootstrap.cc:492] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: Bootstrap starting.
I20260620 15:00:54.402639 9907 catalog_manager.cc:4841] TS 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207): tablet be038e0802194b8489f93ab10848de4d (table test-table [id=223358e5f0bc4e8eac3167892fef3356]) successfully deleted
I20260620 15:00:54.403875 10435 tablet_bootstrap.cc:654] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:54.406190 10432 tablet_bootstrap.cc:492] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: Bootstrap starting.
I20260620 15:00:54.407796 10432 tablet_bootstrap.cc:654] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:54.414672 10435 tablet_bootstrap.cc:492] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: No bootstrap required, opened a new log
I20260620 15:00:54.414907 10435 ts_tablet_manager.cc:1397] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260620 15:00:54.415074 10432 tablet_bootstrap.cc:492] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: No bootstrap required, opened a new log
I20260620 15:00:54.415318 10432 ts_tablet_manager.cc:1397] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260620 15:00:54.415740 10435 raft_consensus.cc:348] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.416078 10435 raft_consensus.cc:374] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:54.416162 10435 raft_consensus.cc:729] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 023a85e9fb7d405d836e8feb8a6b9aa5, State: Initialized, Role: FOLLOWER
I20260620 15:00:54.416438 10435 consensus_queue.cc:260] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.416443 10432 raft_consensus.cc:348] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.416841 10432 raft_consensus.cc:374] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:54.416976 10432 raft_consensus.cc:729] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5abccd676a624f7aa1dd25b15a1ab6c1, State: Initialized, Role: FOLLOWER
I20260620 15:00:54.417666 10435 ts_tablet_manager.cc:1428] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 15:00:54.417303 10432 consensus_queue.cc:260] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.418284 10432 ts_tablet_manager.cc:1428] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260620 15:00:54.419093 10431 tablet_bootstrap.cc:492] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: No bootstrap required, opened a new log
I20260620 15:00:54.419296 10431 ts_tablet_manager.cc:1397] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.012s
I20260620 15:00:54.420181 10431 raft_consensus.cc:348] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.420477 10431 raft_consensus.cc:374] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:54.420569 10431 raft_consensus.cc:729] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378410d6e2894f46848b87c1c20bd3b2, State: Initialized, Role: FOLLOWER
I20260620 15:00:54.421471 10431 consensus_queue.cc:260] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.422290 10431 ts_tablet_manager.cc:1428] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:54.491787 10487 raft_consensus.cc:482] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:54.492046 10487 raft_consensus.cc:504] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.492740 10487 leader_election.cc:288] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207), 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379)
I20260620 15:00:54.493103 10449 raft_consensus.cc:482] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:00:54.493225 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc59179d0461447085c79c0df767680d" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" is_pre_election: true
I20260620 15:00:54.493296 10361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc59179d0461447085c79c0df767680d" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" is_pre_election: true
I20260620 15:00:54.493474 10217 raft_consensus.cc:2454] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 0.
I20260620 15:00:54.493342 10449 raft_consensus.cc:504] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.493965 10010 leader_election.cc:302] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378410d6e2894f46848b87c1c20bd3b2, 5abccd676a624f7aa1dd25b15a1ab6c1; no voters:
I20260620 15:00:54.494037 10449 leader_election.cc:288] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957), 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207)
I20260620 15:00:54.494344 10361 raft_consensus.cc:2454] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 378410d6e2894f46848b87c1c20bd3b2 in term 0.
I20260620 15:00:54.494587 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc59179d0461447085c79c0df767680d" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" is_pre_election: true
I20260620 15:00:54.494565 10079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc59179d0461447085c79c0df767680d" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378410d6e2894f46848b87c1c20bd3b2" is_pre_election: true
I20260620 15:00:54.494784 10079 raft_consensus.cc:2454] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 0.
I20260620 15:00:54.494774 10217 raft_consensus.cc:2454] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 0.
I20260620 15:00:54.495208 10289 leader_election.cc:302] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 023a85e9fb7d405d836e8feb8a6b9aa5, 378410d6e2894f46848b87c1c20bd3b2, 5abccd676a624f7aa1dd25b15a1ab6c1; no voters:
I20260620 15:00:54.495250 10487 raft_consensus.cc:2781] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:54.495340 10487 raft_consensus.cc:482] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:54.495461 10487 raft_consensus.cc:3037] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:54.495697 10449 raft_consensus.cc:2781] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:00:54.495802 10449 raft_consensus.cc:482] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:00:54.495913 10449 raft_consensus.cc:3037] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:54.497929 10449 raft_consensus.cc:504] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.498531 10449 leader_election.cc:288] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 election: Requested vote from peers 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957), 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207)
I20260620 15:00:54.498467 10487 raft_consensus.cc:504] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.498939 10079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc59179d0461447085c79c0df767680d" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "378410d6e2894f46848b87c1c20bd3b2"
I20260620 15:00:54.499104 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc59179d0461447085c79c0df767680d" candidate_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
I20260620 15:00:54.499264 10079 raft_consensus.cc:2379] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in current term 1: Already voted for candidate 378410d6e2894f46848b87c1c20bd3b2 in this term.
I20260620 15:00:54.499279 10217 raft_consensus.cc:3037] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:54.499480 10487 leader_election.cc:288] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 election: Requested vote from peers 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207), 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379)
I20260620 15:00:54.499851 10218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc59179d0461447085c79c0df767680d" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1"
I20260620 15:00:54.499920 10361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc59179d0461447085c79c0df767680d" candidate_uuid: "378410d6e2894f46848b87c1c20bd3b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5"
I20260620 15:00:54.500168 10361 raft_consensus.cc:2379] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 378410d6e2894f46848b87c1c20bd3b2 in current term 1: Already voted for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in this term.
I20260620 15:00:54.500795 10010 leader_election.cc:302] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 378410d6e2894f46848b87c1c20bd3b2; no voters: 023a85e9fb7d405d836e8feb8a6b9aa5, 5abccd676a624f7aa1dd25b15a1ab6c1
I20260620 15:00:54.501513 10438 raft_consensus.cc:2726] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 15:00:54.502143 10217 raft_consensus.cc:2454] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 023a85e9fb7d405d836e8feb8a6b9aa5 in term 1.
I20260620 15:00:54.502485 10289 leader_election.cc:302] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 023a85e9fb7d405d836e8feb8a6b9aa5, 5abccd676a624f7aa1dd25b15a1ab6c1; no voters: 378410d6e2894f46848b87c1c20bd3b2
I20260620 15:00:54.502841 10449 raft_consensus.cc:2781] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:54.503006 10449 raft_consensus.cc:686] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 023a85e9fb7d405d836e8feb8a6b9aa5, State: Running, Role: LEADER
I20260620 15:00:54.503245 10449 consensus_queue.cc:237] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } }
I20260620 15:00:54.506116 9922 catalog_manager.cc:5495] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3). New cstate: current_term: 1 leader_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "023a85e9fb7d405d836e8feb8a6b9aa5" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 41379 } health_report { overall_health: HEALTHY } } }
I20260620 15:00:54.544607 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.539772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.2.70.3:41379)
0620 15:00:54.540049 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.540055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.540082 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:00:54.540244 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.540252 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.540335 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.540430 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:00:54.540809 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.541578 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.542880 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.543323 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.543455 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.543893 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:00:54.544005 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:00:54.544111 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:54.544252 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 15:00:54.544326 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260620 15:00:54.553432 10217 raft_consensus.cc:1261] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Refusing update from remote peer 023a85e9fb7d405d836e8feb8a6b9aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:54.553455 10079 raft_consensus.cc:1261] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Refusing update from remote peer 023a85e9fb7d405d836e8feb8a6b9aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:00:54.554085 10449 consensus_queue.cc:1035] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5abccd676a624f7aa1dd25b15a1ab6c1" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:54.554369 10439 consensus_queue.cc:1035] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "378410d6e2894f46848b87c1c20bd3b2" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 38957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:00:54.609462 9921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:54.609771 9921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55696:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:00:54.613018 9921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f550186d971949a980cd1ef4256f2004: Sending DeleteTablet for 3 replicas of tablet fc59179d0461447085c79c0df767680d
I20260620 15:00:54.613806 10058 tablet_service.cc:1514] Processing DeleteTablet for tablet fc59179d0461447085c79c0df767680d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:56224
I20260620 15:00:54.613966 10197 tablet_service.cc:1514] Processing DeleteTablet for tablet fc59179d0461447085c79c0df767680d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:33954
I20260620 15:00:54.614075 10454 tablet_replica.cc:330] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: stopping tablet replica
I20260620 15:00:54.614225 10455 tablet_replica.cc:330] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: stopping tablet replica
I20260620 15:00:54.614259 10454 raft_consensus.cc:2229] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:54.614394 10455 raft_consensus.cc:2229] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:00:54.614435 10454 raft_consensus.cc:2258] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.614564 10455 raft_consensus.cc:2258] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.615340 10336 tablet_service.cc:1514] Processing DeleteTablet for tablet fc59179d0461447085c79c0df767680d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:00:54 UTC) from {username='oryx', principal='oryx/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:54674
I20260620 15:00:54.615598 10456 tablet_replica.cc:330] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: stopping tablet replica
I20260620 15:00:54.615725 10454 ts_tablet_manager.cc:1905] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.615769 10456 raft_consensus.cc:2229] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:00:54.616052 10456 raft_consensus.cc:2258] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:00:54.617151 10456 ts_tablet_manager.cc:1905] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.619967 10455 ts_tablet_manager.cc:1905] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:00:54.620386 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 9993
I20260620 15:00:54.626574 10454 ts_tablet_manager.cc:1918] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:54.626668 10456 ts_tablet_manager.cc:1918] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:54.626833 10456 log.cc:1196] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/fc59179d0461447085c79c0df767680d
I20260620 15:00:54.626950 10454 log.cc:1196] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/fc59179d0461447085c79c0df767680d
I20260620 15:00:54.626957 10455 ts_tablet_manager.cc:1918] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 15:00:54.627120 10455 log.cc:1196] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/fc59179d0461447085c79c0df767680d
I20260620 15:00:54.627301 10456 ts_tablet_manager.cc:1939] T fc59179d0461447085c79c0df767680d P 023a85e9fb7d405d836e8feb8a6b9aa5: Deleting consensus metadata
I20260620 15:00:54.627543 10455 ts_tablet_manager.cc:1939] T fc59179d0461447085c79c0df767680d P 5abccd676a624f7aa1dd25b15a1ab6c1: Deleting consensus metadata
I20260620 15:00:54.627898 10454 ts_tablet_manager.cc:1939] T fc59179d0461447085c79c0df767680d P 378410d6e2894f46848b87c1c20bd3b2: Deleting consensus metadata
I20260620 15:00:54.628508 9907 catalog_manager.cc:4841] TS 023a85e9fb7d405d836e8feb8a6b9aa5 (127.2.70.3:41379): tablet fc59179d0461447085c79c0df767680d (table test-table [id=364866ff56b44f96b5cc7a1ae5e6eef0]) successfully deleted
I20260620 15:00:54.628893 9907 catalog_manager.cc:4841] TS 5abccd676a624f7aa1dd25b15a1ab6c1 (127.2.70.2:42207): tablet fc59179d0461447085c79c0df767680d (table test-table [id=364866ff56b44f96b5cc7a1ae5e6eef0]) successfully deleted
I20260620 15:00:54.629056 9908 catalog_manager.cc:4841] TS 378410d6e2894f46848b87c1c20bd3b2 (127.2.70.1:38957): tablet fc59179d0461447085c79c0df767680d (table test-table [id=364866ff56b44f96b5cc7a1ae5e6eef0]) successfully deleted
Jun 20 15:00:54 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.1@KRBTEST.COM
I20260620 15:00:54.644788 10426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.621787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56242 (local address 127.2.70.1:38957)
0620 15:00:54.622342 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.622351 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.630003 (+ 7652us) server_negotiation.cc:407] Connection header received
0620 15:00:54.630143 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.630149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.630234 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.630370 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:54.631467 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.632615 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.633943 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.634615 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.634747 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.638612 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:54.638632 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:54.638653 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:54.638697 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:54.642107 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:54.642892 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:54.642899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:54.642901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:54.643017 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:54.643362 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:54.643365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:54.643367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:54.643715 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:54.643911 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:54.644245 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 15:00:54.644417 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260620 15:00:54.645465 10124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:54.787293 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 9993
I20260620 15:00:54.812762 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 10133
Jun 20 15:00:54 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.2@KRBTEST.COM
I20260620 15:00:54.827907 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.813558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33962 (local address 127.2.70.2:42207)
0620 15:00:54.814127 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.814132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.814146 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:00:54.814197 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.814200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.814254 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.814369 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:54.815747 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.816580 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.817851 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.818255 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.818388 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.822546 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:54.822558 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:54.822577 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:54.822613 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:54.825462 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:54.826176 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:54.826183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:54.826186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:54.826314 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:54.826625 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:54.826627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:54.826629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:54.826950 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:54.827166 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:54.827378 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 15:00:54.827546 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":276,"threads_started":1}
I20260620 15:00:54.828532 10263 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:54.925954 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 10133
I20260620 15:00:54.944120 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 10272
Jun 20 15:00:54 dist-test-slave-n5vv krb5kdc[9877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.70.3@KRBTEST.COM
I20260620 15:00:54.958981 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:54.944924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54692 (local address 127.2.70.3:41379)
0620 15:00:54.945151 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:54.945155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:54.945174 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:00:54.945230 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:54.945233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:54.945300 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:54.945383 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:54.946808 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.947569 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.948825 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:54.949215 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:54.949307 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:54.953595 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:54.953616 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:54.953639 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:54.953679 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:54.956413 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:54.957045 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:54.957052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:54.957054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:54.957182 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:54.957691 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:54.957695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:54.957697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:54.958022 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:54.958256 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:54.958463 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 15:00:54.958620 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:00:54.959633 10402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:55.081768 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 10272
I20260620 15:00:55.103205 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 9890
I20260620 15:00:55.104362 9962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:00:55.219398 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 9890
2026-06-20T15:00:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5369 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:55 dist-test-slave-n5vv krb5kdc[10524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:55 dist-test-slave-n5vv krb5kdc[10524](info): set up 2 sockets
Jun 20 15:00:55 dist-test-slave-n5vv krb5kdc[10524](info): commencing operation
krb5kdc: starting...
W20260620 15:00:56.322376 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:56 dist-test-slave-n5vv krb5kdc[10524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:56.540843 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46475
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:46475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:56.861943 10537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:56.862303 10537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:56.862476 10537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:56.869978 10537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:56.870134 10537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:56.870249 10537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:56.870355 10537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:56.870442 10537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:56.880481 10537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:46475
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:56.881942 10537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:56.883278 10537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:56.894775 10542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:56.894716 10543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:56.894806 10545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:56.895953 10537 server_base.cc:1034] running on GCE node
I20260620 15:00:56.896684 10537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:56.898151 10537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:56.899390 10537 hybrid_clock.cc:648] HybridClock initialized: now 1781967656899336 us; error 40 us; skew 500 ppm
I20260620 15:00:56.899755 10537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T15:00:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1747 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:00:57 dist-test-slave-n5vv krb5kdc[10551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:00:57 dist-test-slave-n5vv krb5kdc[10551](info): set up 2 sockets
Jun 20 15:00:57 dist-test-slave-n5vv krb5kdc[10551](info): commencing operation
krb5kdc: starting...
W20260620 15:00:58.070433 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:00:58 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:00:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:00:58.289197 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35577
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:35577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:58.616204 10564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:58.616514 10564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:58.616643 10564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:58.624049 10564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:00:58.624166 10564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:58.624272 10564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:58.624367 10564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:00:58.624426 10564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:00:58.634627 10564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:35577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:58.636102 10564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:58.637441 10564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:58.649636 10569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:58.649608 10570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:58.649717 10572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:58.650458 10564 server_base.cc:1034] running on GCE node
I20260620 15:00:58.651192 10564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:58.652578 10564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:58.653784 10564 hybrid_clock.cc:648] HybridClock initialized: now 1781967658653723 us; error 60 us; skew 500 ppm
I20260620 15:00:58.654142 10564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:58 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967658, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:58.657349 10564 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:00:58.658986 10564 webserver.cc:460] Webserver started at http://127.2.70.62:44287/ using document root <none> and password file <none>
I20260620 15:00:58.659708 10564 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:58.659855 10564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:58.660181 10564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:58.663244 10564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "e489f7197dd44934af927cdce3ccbb7b"
format_stamp: "Formatted at 2026-06-20 15:00:58 on dist-test-slave-n5vv"
server_key: "adc31bad88df1d661af5d67973d140e6"
server_key_iv: "4f429d3cf98eecdd295314cd61219cfb"
server_key_version: "encryptionkey@0"
I20260620 15:00:58.664039 10564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "e489f7197dd44934af927cdce3ccbb7b"
format_stamp: "Formatted at 2026-06-20 15:00:58 on dist-test-slave-n5vv"
server_key: "adc31bad88df1d661af5d67973d140e6"
server_key_iv: "4f429d3cf98eecdd295314cd61219cfb"
server_key_version: "encryptionkey@0"
I20260620 15:00:58.669301 10564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:00:58.672766 10579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:58.674201 10564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:58.674391 10564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "e489f7197dd44934af927cdce3ccbb7b"
format_stamp: "Formatted at 2026-06-20 15:00:58 on dist-test-slave-n5vv"
server_key: "adc31bad88df1d661af5d67973d140e6"
server_key_iv: "4f429d3cf98eecdd295314cd61219cfb"
server_key_version: "encryptionkey@0"
I20260620 15:00:58.674607 10564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:58.702019 10564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:58.706445 10564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:58.706750 10564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:58.724728 10564 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:35577
I20260620 15:00:58.724751 10630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:35577 every 8 connection(s)
I20260620 15:00:58.726168 10564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:00:58.729750 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 10564
I20260620 15:00:58.729983 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:00:58.730126 10631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:00:58.730672 2328 external_mini_cluster.cc:1428] Setting key 87e93187a2f5374c30dffc5359fb6acc
I20260620 15:00:58.743566 10631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Bootstrap starting.
Jun 20 15:00:58 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:58.747514 10631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Neither blocks nor log segments found. Creating new log.
I20260620 15:00:58.748672 10631 log.cc:824] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:00:58.751649 10631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: No bootstrap required, opened a new log
I20260620 15:00:58.753175 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:58.732821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.2.70.62:35577)
0620 15:00:58.733656 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:58.733671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:58.733721 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:00:58.735318 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:58.735352 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:58.735892 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:58.736358 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:58.737680 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:58.738936 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:58.740168 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:58.740908 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:58.741120 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:58.745883 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:58.745892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:58.745908 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:58.745962 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:58.749156 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:58.749965 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:58.749971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:58.749978 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:58.750140 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:58.750587 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:58.750593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:58.750599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:58.751100 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:58.751402 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:58.751823 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 15:00:58.752223 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":339,"threads_started":1}
I20260620 15:00:58.759523 10631 raft_consensus.cc:348] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } }
I20260620 15:00:58.759954 10631 raft_consensus.cc:374] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:00:58.760080 10631 raft_consensus.cc:729] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e489f7197dd44934af927cdce3ccbb7b, State: Initialized, Role: FOLLOWER
I20260620 15:00:58.760695 10631 consensus_queue.cc:260] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } }
I20260620 15:00:58.760941 10631 raft_consensus.cc:388] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:00:58.761096 10631 raft_consensus.cc:482] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:00:58.761268 10631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:00:58.763484 10631 raft_consensus.cc:504] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } }
I20260620 15:00:58.764000 10631 leader_election.cc:302] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e489f7197dd44934af927cdce3ccbb7b; no voters:
I20260620 15:00:58.764597 10631 leader_election.cc:288] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:00:58.764726 10636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:00:58.765450 10636 raft_consensus.cc:686] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 1 LEADER]: Becoming Leader. State: Replica: e489f7197dd44934af927cdce3ccbb7b, State: Running, Role: LEADER
I20260620 15:00:58.765967 10636 consensus_queue.cc:237] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } }
I20260620 15:00:58.766708 10631 sys_catalog.cc:564] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:00:58.768919 10638 sys_catalog.cc:455] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e489f7197dd44934af927cdce3ccbb7b. Latest consensus state: current_term: 1 leader_uuid: "e489f7197dd44934af927cdce3ccbb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } } }
I20260620 15:00:58.769361 10638 sys_catalog.cc:458] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:58.772151 10637 sys_catalog.cc:455] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e489f7197dd44934af927cdce3ccbb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } } }
I20260620 15:00:58.772408 10637 sys_catalog.cc:458] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: This master's current role is: LEADER
I20260620 15:00:58.772965 10644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:00:58.778318 10644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:00:58.779956 10653 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:00:58.787443 10644 catalog_manager.cc:1347] Generated new cluster ID: f5624b5ad0ae4d43a1e1c14b578f1fa9
I20260620 15:00:58.787563 10644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:00:58.800431 10644 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:00:58.802382 10644 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:00:58.814802 10644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Generated new TSK 0
I20260620 15:00:58.815917 10644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:00:58.874197 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35577
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:59.207875 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:59.208158 10657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:59.208329 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:59.215672 10657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:59.215806 10657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:59.215977 10657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:00:59.225713 10657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:59.227027 10657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:59.228319 10657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:59.242440 10665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:59.242591 10663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:59.242545 10662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:59.247047 10657 server_base.cc:1034] running on GCE node
I20260620 15:00:59.247691 10657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:59.248699 10657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:59.249898 10657 hybrid_clock.cc:648] HybridClock initialized: now 1781967659249831 us; error 57 us; skew 500 ppm
I20260620 15:00:59.250224 10657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:59 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967659, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:59.257915 10657 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:00:59.259495 10657 webserver.cc:460] Webserver started at http://127.2.70.1:41833/ using document root <none> and password file <none>
I20260620 15:00:59.260226 10657 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:59.260372 10657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:59.260685 10657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:59.263711 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "7a1597f34ec843c9b5080e560df246b0"
format_stamp: "Formatted at 2026-06-20 15:00:59 on dist-test-slave-n5vv"
server_key: "7f4a284e2830b45303b9d7874593ed06"
server_key_iv: "f61dfee1ca28f17375da1b8151a41808"
server_key_version: "encryptionkey@0"
I20260620 15:00:59.264565 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "7a1597f34ec843c9b5080e560df246b0"
format_stamp: "Formatted at 2026-06-20 15:00:59 on dist-test-slave-n5vv"
server_key: "7f4a284e2830b45303b9d7874593ed06"
server_key_iv: "f61dfee1ca28f17375da1b8151a41808"
server_key_version: "encryptionkey@0"
I20260620 15:00:59.269920 10657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:00:59.273552 10672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:59.274909 10657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:00:59.275079 10657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "7a1597f34ec843c9b5080e560df246b0"
format_stamp: "Formatted at 2026-06-20 15:00:59 on dist-test-slave-n5vv"
server_key: "7f4a284e2830b45303b9d7874593ed06"
server_key_iv: "f61dfee1ca28f17375da1b8151a41808"
server_key_version: "encryptionkey@0"
I20260620 15:00:59.275285 10657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:59.300542 10657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:59.304642 10657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:59.304970 10657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:59.305974 10657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:59.307291 10657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:59.307395 10657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:59.307546 10657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:59.307618 10657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:59.340397 10657 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:41723
I20260620 15:00:59.340446 10784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:41723 every 8 connection(s)
I20260620 15:00:59.341815 10657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:00:59.346015 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 10657
I20260620 15:00:59.346237 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:00:59.346858 2328 external_mini_cluster.cc:1428] Setting key 55600264021a9e792993fdad6fb9c72c
Jun 20 15:00:59 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967659, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:59.361837 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:59.344327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:58933 (local address 127.2.70.62:35577)
0620 15:00:59.345161 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:59.345169 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:59.346022 (+ 853us) server_negotiation.cc:407] Connection header received
0620 15:00:59.348058 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:59.348063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:59.348162 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:59.348277 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:59.349799 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:59.350521 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:59.352079 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:59.352478 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:59.352614 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:59.356270 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:59.356275 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:59.356279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:59.356310 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:59.358842 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:59.359549 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:59.359558 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:59.359560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:59.359669 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:59.360088 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:59.360093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:59.360097 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:59.360347 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:59.360508 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:59.361198 (+ 690us) server_negotiation.cc:299] Negotiation successful
0620 15:00:59.361425 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":583,"threads_started":1}
I20260620 15:00:59.362826 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:59.344942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:58933)
0620 15:00:59.345858 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:59.345899 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:59.347784 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:59.348510 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:59.348520 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:59.349009 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:59.349634 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:59.349652 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:59.350679 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:59.350685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:59.351910 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:59.351921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:59.352703 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:59.352712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:59.352954 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:59.353691 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:59.353713 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:59.356045 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:59.358984 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:59.358994 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:59.359011 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:59.359413 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:59.359775 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:59.359781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:59.359785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:59.359971 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:59.360490 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:59.360507 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:59.360836 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 15:00:59.361097 (+ 261us) client_negotiation.cc:240] Negotiation successful
0620 15:00:59.361691 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":302,"threads_started":1}
I20260620 15:00:59.364450 10785 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35577
I20260620 15:00:59.364769 10785 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:59.365471 10785 heartbeater.cc:511] Master 127.2.70.62:35577 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:00:59.367694 10596 ts_manager.cc:194] Registered new tserver with Master: 7a1597f34ec843c9b5080e560df246b0 (127.2.70.1:41723)
I20260620 15:00:59.369220 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:58933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:00:59.394794 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35577
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:00:59.719025 10791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:00:59.719297 10791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:00:59.719466 10791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:00:59.726842 10791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:00:59.726951 10791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:00:59.727128 10791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:00:59.736874 10791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:00:59.738127 10791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:00:59.739459 10791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:00:59.758805 10797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:00:59.758996 10796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:59.759856 10791 server_base.cc:1034] running on GCE node
W20260620 15:00:59.761737 10799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:00:59.762717 10791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:00:59.764087 10791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:00:59.765321 10791 hybrid_clock.cc:648] HybridClock initialized: now 1781967659765237 us; error 63 us; skew 500 ppm
I20260620 15:00:59.765749 10791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:00:59 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967659, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:00:59.770213 10791 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:00:59.772110 10791 webserver.cc:460] Webserver started at http://127.2.70.2:40921/ using document root <none> and password file <none>
I20260620 15:00:59.773056 10791 fs_manager.cc:359] Metadata directory not provided
I20260620 15:00:59.773218 10791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:00:59.773638 10791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:00:59.777737 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "73e86e6f2dcb4a759df7fbe17156c768"
format_stamp: "Formatted at 2026-06-20 15:00:59 on dist-test-slave-n5vv"
server_key: "c557338b98ce6dfab87dc8f6a830600a"
server_key_iv: "ff76b27e0b497f158a11655fcf78bec7"
server_key_version: "encryptionkey@0"
I20260620 15:00:59.778790 10791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "73e86e6f2dcb4a759df7fbe17156c768"
format_stamp: "Formatted at 2026-06-20 15:00:59 on dist-test-slave-n5vv"
server_key: "c557338b98ce6dfab87dc8f6a830600a"
server_key_iv: "ff76b27e0b497f158a11655fcf78bec7"
server_key_version: "encryptionkey@0"
I20260620 15:00:59.785826 10791 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260620 15:00:59.789538 10806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:59.791056 10791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:00:59.791231 10791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "73e86e6f2dcb4a759df7fbe17156c768"
format_stamp: "Formatted at 2026-06-20 15:00:59 on dist-test-slave-n5vv"
server_key: "c557338b98ce6dfab87dc8f6a830600a"
server_key_iv: "ff76b27e0b497f158a11655fcf78bec7"
server_key_version: "encryptionkey@0"
I20260620 15:00:59.791435 10791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:00:59.813182 10791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:00:59.817255 10791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:00:59.817585 10791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:00:59.818512 10791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:00:59.819784 10791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:00:59.819895 10791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:59.820043 10791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:00:59.820115 10791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:00:59.853178 10791 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:37339
I20260620 15:00:59.853233 10918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:37339 every 8 connection(s)
I20260620 15:00:59.854558 10791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:00:59.855860 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 10791
I20260620 15:00:59.856050 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:00:59.856487 2328 external_mini_cluster.cc:1428] Setting key ef7d19a1b2e447d09257e2dc821a4a20
Jun 20 15:00:59 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967659, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:00:59.878166 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:59.857334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:56999 (local address 127.2.70.62:35577)
0620 15:00:59.857551 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:00:59.857557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:00:59.858925 (+ 1368us) server_negotiation.cc:407] Connection header received
0620 15:00:59.861380 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:00:59.861385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:00:59.861475 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:00:59.861608 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:00:59.863221 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:59.864082 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:59.866311 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:00:59.867100 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:00:59.867243 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:59.870781 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:00:59.870790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:00:59.870798 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:00:59.870844 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:00:59.874625 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:59.875425 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:59.875431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:59.875436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:59.875596 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:00:59.876176 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:00:59.876184 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:00:59.876190 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:00:59.876596 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:00:59.876757 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:00:59.877511 (+ 754us) server_negotiation.cc:299] Negotiation successful
0620 15:00:59.877760 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:00:59.879202 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:00:59.857915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:56999)
0620 15:00:59.858770 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 15:00:59.858815 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:00:59.861065 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:00:59.861879 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:00:59.861888 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:00:59.862420 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:00:59.863044 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:59.863062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:59.864242 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:59.864248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:59.866168 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:00:59.866179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:00:59.867309 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:00:59.867315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:00:59.867605 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:00:59.868354 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:00:59.868378 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:00:59.870543 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:00:59.874759 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:59.874767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:59.874781 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:59.875264 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:59.875746 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:00:59.875753 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:00:59.875759 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:00:59.876027 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:00:59.876744 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:00:59.876750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:00:59.877101 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 15:00:59.877329 (+ 228us) client_negotiation.cc:240] Negotiation successful
0620 15:00:59.877894 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":346,"threads_started":1}
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:00:59.880729 10919 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35577
I20260620 15:00:59.881003 10919 heartbeater.cc:464] Registering TS with master...
I20260620 15:00:59.881819 10919 heartbeater.cc:511] Master 127.2.70.62:35577 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
I20260620 15:00:59.883373 10596 ts_manager.cc:194] Registered new tserver with Master: 73e86e6f2dcb4a759df7fbe17156c768 (127.2.70.2:37339)
I20260620 15:00:59.884663 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:56999
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:00:59.908720 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35577
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:00.235678 10924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:00.236008 10924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:00.236164 10924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:00.243593 10924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:00.243762 10924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:00.243945 10924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:01:00.253813 10924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:00.255115 10924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:00.256440 10924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:00.269831 10929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:00.269876 10932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:00.269791 10930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:00.270798 10924 server_base.cc:1034] running on GCE node
I20260620 15:01:00.271392 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:00.272341 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:00.273613 10924 hybrid_clock.cc:648] HybridClock initialized: now 1781967660273552 us; error 42 us; skew 500 ppm
I20260620 15:01:00.273995 10924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:00 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967660, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:00.276993 10924 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:01:00.278570 10924 webserver.cc:460] Webserver started at http://127.2.70.3:32813/ using document root <none> and password file <none>
I20260620 15:01:00.279367 10924 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:00.279510 10924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:00.279826 10924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:00.282763 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "fbbba502d2734243919710e327f55d61"
format_stamp: "Formatted at 2026-06-20 15:01:00 on dist-test-slave-n5vv"
server_key: "fe2f099cfac382fa63f2aa12e323b2b6"
server_key_iv: "8fe7fe9d671959a7088e126b1a3a07ad"
server_key_version: "encryptionkey@0"
I20260620 15:01:00.283536 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "fbbba502d2734243919710e327f55d61"
format_stamp: "Formatted at 2026-06-20 15:01:00 on dist-test-slave-n5vv"
server_key: "fe2f099cfac382fa63f2aa12e323b2b6"
server_key_iv: "8fe7fe9d671959a7088e126b1a3a07ad"
server_key_version: "encryptionkey@0"
I20260620 15:01:00.288605 10924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 15:01:00.292063 10939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:00.293339 10924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:01:00.293596 10924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "fbbba502d2734243919710e327f55d61"
format_stamp: "Formatted at 2026-06-20 15:01:00 on dist-test-slave-n5vv"
server_key: "fe2f099cfac382fa63f2aa12e323b2b6"
server_key_iv: "8fe7fe9d671959a7088e126b1a3a07ad"
server_key_version: "encryptionkey@0"
I20260620 15:01:00.293803 10924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:00.327543 10924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:00.331622 10924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:00.331919 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:00.332870 10924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:00.334261 10924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:00.334364 10924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:00.334517 10924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:00.334587 10924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:00.368289 10924 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:42773
I20260620 15:01:00.368325 11051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:42773 every 8 connection(s)
I20260620 15:01:00.369634 10924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:00.372221 10785 heartbeater.cc:503] Master 127.2.70.62:35577 was elected leader, sending a full tablet report...
I20260620 15:01:00.379904 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 10924
I20260620 15:01:00.380136 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:00.380563 2328 external_mini_cluster.cc:1428] Setting key d40523b6d0e9a8d049d88038c909989c
Jun 20 15:01:00 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967660, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:01:00.389736 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:00.372152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:48561 (local address 127.2.70.62:35577)
0620 15:01:00.372338 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:00.372344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:00.373848 (+ 1504us) server_negotiation.cc:407] Connection header received
0620 15:01:00.375624 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:00.375627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:00.375688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:00.375772 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:00.377200 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.378110 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.379841 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.380401 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.380539 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:00.384446 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:00.384452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:00.384455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:00.384487 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:00.387159 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:00.387786 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:00.387790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:00.387792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:00.387906 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:00.388253 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:00.388256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:00.388258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:00.388500 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:00.388616 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:00.389247 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 15:01:00.389399 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:01:00.390762 11054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:00.372847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:48561)
0620 15:01:00.373710 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:00.373756 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:00.375388 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:00.375991 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:00.376013 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:00.376501 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:00.377063 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:00.377081 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.378258 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.378264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:00.379701 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:00.379713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.380639 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.380646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:00.380887 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:00.381662 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:00.381703 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:00.384264 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:00.387278 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:00.387285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:00.387298 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:00.387687 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:00.388002 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:00.388005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:00.388007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:00.388166 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:00.388614 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:00.388620 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:00.388921 (+ 301us) client_negotiation.cc:769] Sending connection context
0620 15:01:00.389163 (+ 242us) client_negotiation.cc:240] Negotiation successful
0620 15:01:00.389607 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":364,"threads_started":1}
I20260620 15:01:00.392159 11052 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35577
I20260620 15:01:00.392472 11052 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:00.393190 11052 heartbeater.cc:511] Master 127.2.70.62:35577 requested a full tablet report, sending...
I20260620 15:01:00.394734 10596 ts_manager.cc:194] Registered new tserver with Master: fbbba502d2734243919710e327f55d61 (127.2.70.3:42773)
I20260620 15:01:00.395852 10596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:48561
I20260620 15:01:00.395938 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab.
Entry for principal oryx/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab.
WARNING: no policy specified for oryx/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab.
Entry for principal oryx/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab.
WARNING: no policy specified for oryx/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab.
Entry for principal oryx/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab.
I20260620 15:01:00.536490 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 10657
Jun 20 15:01:00 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:01:00.553239 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:00.537665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.2.70.1:41723)
0620 15:01:00.538377 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:00.538384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:00.538412 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:01:00.538476 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:00.538482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:00.538735 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:00.538885 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:00.540164 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.541251 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.542627 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.543224 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.543333 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:00.547579 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:00.547591 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:00.547606 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:00.547645 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:00.550413 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:00.551115 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:00.551123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:00.551125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:00.551256 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:00.551647 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:00.551651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:00.551654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:00.551985 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:00.552290 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:00.552610 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:01:00.552888 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":383,"threads_started":1}
I20260620 15:01:00.554553 10780 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:00.637881 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 10657
I20260620 15:01:00.654086 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 10791
Jun 20 15:01:00 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:01:00.670885 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:00.654959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.2.70.2:37339)
0620 15:01:00.655594 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:00.655602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:00.655629 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:01:00.655690 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:00.655697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:00.655909 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:00.656049 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:00.657199 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.658304 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.659742 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.660311 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.660416 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:00.664772 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:00.664784 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:00.664799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:00.664842 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:00.667851 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:00.668624 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:00.668631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:00.668634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:00.668758 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:00.669117 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:00.669123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:00.669126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:00.669460 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:00.669956 (+ 496us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:00.670261 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 15:01:00.670528 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":330,"threads_started":1}
I20260620 15:01:00.672160 10914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:00.754733 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 10791
I20260620 15:01:00.770658 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 10924
Jun 20 15:01:00 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967658, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:01:00.786842 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:00.771687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53190 (local address 127.2.70.3:42773)
0620 15:01:00.772348 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:00.772355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:00.772390 (+ 35us) server_negotiation.cc:407] Connection header received
0620 15:01:00.772460 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:00.772466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:00.772736 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:00.772912 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:00.774218 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.775215 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.776387 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:00.776924 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:00.777017 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:00.781453 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:00.781464 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:00.781623 (+ 159us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:00.781656 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:00.784435 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:00.785018 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:00.785025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:00.785027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:00.785147 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:00.785433 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:00.785436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:00.785438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:00.785806 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:00.786055 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:00.786273 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 15:01:00.786460 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":359,"threads_started":1}
I20260620 15:01:00.788098 11047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:00.870697 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 10924
I20260620 15:01:00.887034 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 10564
I20260620 15:01:00.888254 10626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:00.989833 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 10564
I20260620 15:01:01.010210 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35577
--webserver_interface=127.2.70.62
--webserver_port=44287
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:35577
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:01.336947 11069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:01.337256 11069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:01.337422 11069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:01.344801 11069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:01.344911 11069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:01.345019 11069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:01.345121 11069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:01.345204 11069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:01.355259 11069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:35577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=44287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:01.356617 11069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:01.357972 11069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:01.370083 11075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:01.370146 11077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:01.370174 11074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:01.370831 11069 server_base.cc:1034] running on GCE node
I20260620 15:01:01.371634 11069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:01.373157 11069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:01.374385 11069 hybrid_clock.cc:648] HybridClock initialized: now 1781967661374335 us; error 67 us; skew 500 ppm
I20260620 15:01:01.374735 11069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:01 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967661, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:01.377851 11069 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:01:01.379485 11069 webserver.cc:460] Webserver started at http://127.2.70.62:44287/ using document root <none> and password file <none>
I20260620 15:01:01.380205 11069 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:01.380367 11069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:01.386107 11069 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:01:01.389060 11084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:01.390475 11069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:01.390664 11069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "e489f7197dd44934af927cdce3ccbb7b"
format_stamp: "Formatted at 2026-06-20 15:00:58 on dist-test-slave-n5vv"
server_key: "adc31bad88df1d661af5d67973d140e6"
server_key_iv: "4f429d3cf98eecdd295314cd61219cfb"
server_key_version: "encryptionkey@0"
I20260620 15:01:01.391634 11069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:01.420676 11069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:01.425125 11069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:01.425415 11069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:01.444260 11069 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:35577
I20260620 15:01:01.444296 11135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:35577 every 8 connection(s)
I20260620 15:01:01.445974 11069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:01.453855 11136 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:01:01.454020 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 11069
I20260620 15:01:01.454784 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:41723
--local_ip_for_outbound_sockets=127.2.70.1
--tserver_master_addrs=127.2.70.62:35577
--webserver_port=41833
--webserver_interface=127.2.70.1
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.70.1
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:01:01.457705 11136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Bootstrap starting.
I20260620 15:01:01.478461 11136 log.cc:824] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:01.488027 11136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:01:01.488631 11136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Bootstrap complete.
I20260620 15:01:01.498963 11136 raft_consensus.cc:348] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } }
I20260620 15:01:01.499887 11136 raft_consensus.cc:729] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e489f7197dd44934af927cdce3ccbb7b, State: Initialized, Role: FOLLOWER
I20260620 15:01:01.500522 11136 consensus_queue.cc:260] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } }
I20260620 15:01:01.500792 11136 raft_consensus.cc:388] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:01.500937 11136 raft_consensus.cc:482] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:01.501132 11136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:01:01.505290 11136 raft_consensus.cc:504] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } }
I20260620 15:01:01.505890 11136 leader_election.cc:302] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e489f7197dd44934af927cdce3ccbb7b; no voters:
I20260620 15:01:01.506654 11136 leader_election.cc:288] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:01:01.506798 11140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:01:01.507947 11140 raft_consensus.cc:686] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [term 2 LEADER]: Becoming Leader. State: Replica: e489f7197dd44934af927cdce3ccbb7b, State: Running, Role: LEADER
I20260620 15:01:01.508540 11140 consensus_queue.cc:237] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } }
I20260620 15:01:01.509084 11136 sys_catalog.cc:564] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:01.511358 11141 sys_catalog.cc:455] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e489f7197dd44934af927cdce3ccbb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } } }
I20260620 15:01:01.511865 11141 sys_catalog.cc:458] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:01.514204 11142 sys_catalog.cc:455] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e489f7197dd44934af927cdce3ccbb7b. Latest consensus state: current_term: 2 leader_uuid: "e489f7197dd44934af927cdce3ccbb7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e489f7197dd44934af927cdce3ccbb7b" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35577 } } }
I20260620 15:01:01.514591 11142 sys_catalog.cc:458] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:01.516089 11146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:01.522311 11146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:01.523768 11146 catalog_manager.cc:1259] Loaded cluster ID: f5624b5ad0ae4d43a1e1c14b578f1fa9
I20260620 15:01:01.523869 11146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:01.528263 11146 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:01.529881 11146 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e489f7197dd44934af927cdce3ccbb7b: Loaded TSK: 0
I20260620 15:01:01.531529 11146 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:01:01.802615 11138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:01.802958 11138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:01.803121 11138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:01.810750 11138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:01.810921 11138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:01.811117 11138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:01:01.821177 11138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.1.keytab
--principal=oryx/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:41723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=41833
--tserver_master_addrs=127.2.70.62:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:01.822656 11138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:01.824035 11138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:01.838810 11162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:01.838871 11163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:01.840111 11165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:01.841073 11138 server_base.cc:1034] running on GCE node
I20260620 15:01:01.841820 11138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:01.842845 11138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:01.844080 11138 hybrid_clock.cc:648] HybridClock initialized: now 1781967661844030 us; error 43 us; skew 500 ppm
I20260620 15:01:01.844398 11138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:01 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967661, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:01.847808 11138 init.cc:377] Logged in from keytab as oryx/127.2.70.1@KRBTEST.COM (short username oryx)
I20260620 15:01:01.849366 11138 webserver.cc:460] Webserver started at http://127.2.70.1:41833/ using document root <none> and password file <none>
I20260620 15:01:01.850106 11138 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:01.850262 11138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:01.855942 11138 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:01:01.858901 11172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:01.860311 11138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:01.860499 11138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "7a1597f34ec843c9b5080e560df246b0"
format_stamp: "Formatted at 2026-06-20 15:00:59 on dist-test-slave-n5vv"
server_key: "7f4a284e2830b45303b9d7874593ed06"
server_key_iv: "f61dfee1ca28f17375da1b8151a41808"
server_key_version: "encryptionkey@0"
I20260620 15:01:01.861500 11138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:01.884876 11138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:01.888955 11138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:01.889258 11138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:01.890290 11138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:01.891567 11138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:01.891662 11138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:01.891739 11138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:01.891793 11138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:01.924830 11138 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:41723
I20260620 15:01:01.924921 11284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:41723 every 8 connection(s)
I20260620 15:01:01.926288 11138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:01.927392 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 11138
I20260620 15:01:01.928136 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:37339
--local_ip_for_outbound_sockets=127.2.70.2
--tserver_master_addrs=127.2.70.62:35577
--webserver_port=40921
--webserver_interface=127.2.70.2
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.70.2
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:01:01 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:01 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:01:01.948354 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:01.929920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:56595 (local address 127.2.70.62:35577)
0620 15:01:01.930676 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:01.930693 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:01.931487 (+ 794us) server_negotiation.cc:407] Connection header received
0620 15:01:01.934662 (+ 3175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:01.934676 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:01.935106 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:01.935547 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:01.936893 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.938941 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.940541 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.941723 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.941911 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:01.947385 (+ 5474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:56595: BlockingRecv error: recv got EOF from 127.2.70.1:56595 (error 108)
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":271,"threads_started":1}
I20260620 15:01:01.948482 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:01.930035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:56595)
0620 15:01:01.931278 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:01.931330 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:01.934084 (+ 2754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:01.935658 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:01.935665 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:01.936158 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:01.936699 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:01.936715 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.939095 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.939100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:01.940393 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:01.940404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.941922 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.941927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:01.942112 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:01.942794 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:01.942816 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:01.947094 (+ 4278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":933,"thread_start_us":748,"threads_started":1}
W20260620 15:01:01.949214 11285 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:35577 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.70.62:35577: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Jun 20 15:01:01 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:01 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:01.957856 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:01.949997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:43599)
0620 15:01:01.950183 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:01.950203 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:01.950318 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:01.950654 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:01.950659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:01.951044 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:01.951177 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:01.951185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.952175 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.952182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:01.953068 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:01.953078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.953812 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.953817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:01.953987 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:01.954577 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:01.954594 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:01.957558 (+ 2964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 15:01:01.958004 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:01.950082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:43599 (local address 127.2.70.62:35577)
0620 15:01:01.950233 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:01.950240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:01.950259 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:01:01.950442 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:01.950447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:01.950515 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:01.950631 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:01.951312 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.952036 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.953203 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.953671 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.953800 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:01.957778 (+ 3978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:43599: BlockingRecv error: recv got EOF from 127.2.70.1:43599 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 15:01:01 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:01 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:01.966666 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:01.958555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:39645)
0620 15:01:01.958738 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:01.958757 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:01.958877 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:01.959256 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:01.959260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:01.959518 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:01.959650 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:01.959657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.961054 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.961057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:01.962045 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:01.962056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.962957 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.962961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:01.963092 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:01.963687 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:01.963704 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:01.966452 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:01:01.966948 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:01.958770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:39645 (local address 127.2.70.62:35577)
0620 15:01:01.958948 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:01.958954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:01.958972 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:01:01.959020 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:01.959025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:01.959105 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:01.959205 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:01.959792 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.960904 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.962173 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:01.962820 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:01.962946 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:01.966684 (+ 3738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:39645: BlockingRecv error: recv got EOF from 127.2.70.1:39645 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 15:01:01.966995 11285 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:01:02.270843 11289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:02.271173 11289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:02.271267 11289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:02.278467 11289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:02.278590 11289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:02.278704 11289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:01:02.288395 11289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.2.keytab
--principal=oryx/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:37339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=40921
--tserver_master_addrs=127.2.70.62:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:02.289707 11289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:02.290935 11289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:02.303834 11294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:02.303764 11295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:02.303799 11297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:02.304538 11289 server_base.cc:1034] running on GCE node
I20260620 15:01:02.305075 11289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:02.306015 11289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:02.307211 11289 hybrid_clock.cc:648] HybridClock initialized: now 1781967662307149 us; error 53 us; skew 500 ppm
I20260620 15:01:02.307476 11289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:02.310645 11289 init.cc:377] Logged in from keytab as oryx/127.2.70.2@KRBTEST.COM (short username oryx)
I20260620 15:01:02.312031 11289 webserver.cc:460] Webserver started at http://127.2.70.2:40921/ using document root <none> and password file <none>
I20260620 15:01:02.312718 11289 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:02.312868 11289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:02.318279 11289 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:01:02.321456 11304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:02.322924 11289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:01:02.323109 11289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "73e86e6f2dcb4a759df7fbe17156c768"
format_stamp: "Formatted at 2026-06-20 15:00:59 on dist-test-slave-n5vv"
server_key: "c557338b98ce6dfab87dc8f6a830600a"
server_key_iv: "ff76b27e0b497f158a11655fcf78bec7"
server_key_version: "encryptionkey@0"
I20260620 15:01:02.324093 11289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:02.342916 11289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:02.346729 11289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:02.347007 11289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:02.347968 11289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:02.349249 11289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:02.349375 11289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:02.349545 11289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:02.349640 11289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:02.381528 11289 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:37339
I20260620 15:01:02.381582 11416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:37339 every 8 connection(s)
I20260620 15:01:02.383095 11289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:02.393406 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 11289
I20260620 15:01:02.394045 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:42773
--local_ip_for_outbound_sockets=127.2.70.3
--tserver_master_addrs=127.2.70.62:35577
--webserver_port=32813
--webserver_interface=127.2.70.3
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.70.3
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:01:02.399355 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:02.385712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:52017 (local address 127.2.70.62:35577)
0620 15:01:02.385881 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:02.385885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:02.386981 (+ 1096us) server_negotiation.cc:407] Connection header received
0620 15:01:02.388817 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:02.388823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:02.388910 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:02.389019 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:02.390543 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.391232 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.392790 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.393437 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.393600 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.399172 (+ 5572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:52017: BlockingRecv error: recv got EOF from 127.2.70.2:52017 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:01:02.400324 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:02.386147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:52017)
0620 15:01:02.386825 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:02.386863 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:02.388535 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:02.389255 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:02.389263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:02.389832 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:02.390373 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.390389 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.391408 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.391412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.392631 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.392642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.393696 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.393702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.393891 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.394610 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:02.394636 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:02.399016 (+ 4380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":271,"threads_started":1}
W20260620 15:01:02.401067 11417 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:35577 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.70.62:35577: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:02.409557 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:02.401680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:51589)
0620 15:01:02.401850 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:02.401869 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:02.401980 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:02.402329 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:02.402333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:02.402650 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:02.402777 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.402784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.403749 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.403754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.404868 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.404897 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.405631 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.405636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.405818 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.406384 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:02.406400 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:02.409232 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:01:02.409693 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:02.401748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:51589 (local address 127.2.70.62:35577)
0620 15:01:02.401903 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:02.401911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:02.401928 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:01:02.402131 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:02.402136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:02.402207 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:02.402309 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:02.402910 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.403607 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.405047 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.405443 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.405620 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.409430 (+ 3810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:51589: BlockingRecv error: recv got EOF from 127.2.70.2:51589 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:02.419993 11419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:02.410348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:34017)
0620 15:01:02.410574 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:02.410596 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:02.410698 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:02.411064 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:02.411070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:02.411496 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:02.411732 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.411748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.412697 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.412704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.414474 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.414493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.415194 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.415202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.415425 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.416029 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:02.416057 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:02.419701 (+ 3644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:01:02.420228 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:02.410557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:34017 (local address 127.2.70.62:35577)
0620 15:01:02.410746 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:02.410749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:02.410765 (+ 16us) server_negotiation.cc:407] Connection header received
0620 15:01:02.410835 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:02.410841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:02.410915 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:02.411008 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:02.411885 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.412562 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.414634 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.415044 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.415143 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.419879 (+ 4736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:34017: BlockingRecv error: recv got EOF from 127.2.70.2:34017 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 15:01:02.420279 11417 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:01:02.726994 11420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:02.727336 11420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:02.727433 11420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:02.734608 11420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:02.734721 11420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:02.734833 11420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:01:02.744411 11420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/krb5kdc/oryx_127.2.70.3.keytab
--principal=oryx/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:42773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=32813
--tserver_master_addrs=127.2.70.62:35577
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:02.745692 11420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:02.746954 11420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:02.760485 11426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:02.760452 11428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:02.760561 11425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:02.762923 11420 server_base.cc:1034] running on GCE node
I20260620 15:01:02.763548 11420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:02.764487 11420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:02.765714 11420 hybrid_clock.cc:648] HybridClock initialized: now 1781967662765675 us; error 44 us; skew 500 ppm
I20260620 15:01:02.766094 11420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967662, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:02.769459 11420 init.cc:377] Logged in from keytab as oryx/127.2.70.3@KRBTEST.COM (short username oryx)
I20260620 15:01:02.771036 11420 webserver.cc:460] Webserver started at http://127.2.70.3:32813/ using document root <none> and password file <none>
I20260620 15:01:02.771744 11420 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:02.771894 11420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:02.777257 11420 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:01:02.779919 11435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:02.781209 11420 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:01:02.781385 11420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "fbbba502d2734243919710e327f55d61"
format_stamp: "Formatted at 2026-06-20 15:01:00 on dist-test-slave-n5vv"
server_key: "fe2f099cfac382fa63f2aa12e323b2b6"
server_key_iv: "8fe7fe9d671959a7088e126b1a3a07ad"
server_key_version: "encryptionkey@0"
I20260620 15:01:02.782387 11420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:02.810629 11420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:02.814685 11420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:02.814954 11420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:02.815924 11420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:02.817277 11420 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:02.817401 11420 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:02.817550 11420 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:02.817636 11420 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:02.851995 11420 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:42773
I20260620 15:01:02.852054 11547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:42773 every 8 connection(s)
I20260620 15:01:02.853597 11420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:02.859237 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 11420
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:01:02.870126 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:02.856197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:50993 (local address 127.2.70.62:35577)
0620 15:01:02.856370 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:02.856375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:02.857669 (+ 1294us) server_negotiation.cc:407] Connection header received
0620 15:01:02.859464 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:02.859470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:02.859576 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:02.859689 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:02.861604 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.862320 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.863945 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.864349 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.864481 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.869815 (+ 5334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:50993: BlockingRecv error: recv got EOF from 127.2.70.3:50993 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:01:02.870949 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:02.856762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:50993)
0620 15:01:02.857449 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:02.857543 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:02.859172 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:02.859939 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:02.859948 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:02.860773 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:02.861406 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.861424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.862485 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.862498 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.863780 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.863793 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.864575 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.864582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.864812 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.865583 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:02.865606 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:02.869701 (+ 4095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":279,"threads_started":1}
W20260620 15:01:02.871799 11548 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:35577 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.70.62:35577: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
I20260620 15:01:02.876173 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:02.860231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.2.70.62:35577)
0620 15:01:02.860974 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:02.860980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:02.860995 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:01:02.861048 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:02.861051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:02.861100 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:02.861211 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:02.862790 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.863550 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.864843 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.865460 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.865620 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.868596 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:02.868606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:02.868624 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:02.868666 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:02.872411 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:02.873461 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:02.873468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:02.873475 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:02.873684 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:02.874111 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:02.874118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:02.874127 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:02.874591 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:02.874925 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:02.875373 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 15:01:02.875742 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":498,"threads_started":1}
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:02.881110 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:02.872485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:52505)
0620 15:01:02.872665 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:02.872685 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:02.872793 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:02.873103 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:02.873109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:02.873583 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:02.873747 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.873761 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.874808 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.874815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.875786 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.875800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.876708 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.876714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.876902 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.877566 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:02.877588 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:02.880828 (+ 3240us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 15:01:02.881260 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:02.872614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:52505 (local address 127.2.70.62:35577)
0620 15:01:02.872771 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:02.872775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:02.872790 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:01:02.872913 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:02.872916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:02.872968 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:02.873047 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:02.873896 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.874666 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.875926 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.876563 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.876661 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.881023 (+ 4362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:52505: BlockingRecv error: recv got EOF from 127.2.70.3:52505 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:02.889643 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:02.881952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:43925)
0620 15:01:02.882101 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:02.882116 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:02.882235 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:02.882543 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:02.882548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:02.882833 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:02.882972 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.882980 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.883974 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.883980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.884905 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.884916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.885733 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.885739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.885911 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.886618 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:02.886636 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:02.889345 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:01:02.889832 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:02.882034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:43925 (local address 127.2.70.62:35577)
0620 15:01:02.882261 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:02.882268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:02.882288 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:01:02.882336 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:02.882340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:02.882405 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:02.882504 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:02.883107 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.883839 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.885060 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.885591 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.885723 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.889587 (+ 3864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:43925: BlockingRecv error: recv got EOF from 127.2.70.3:43925 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260620 15:01:02.889961 11548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:02 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:02.975476 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:02.967537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:44615)
0620 15:01:02.968053 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:02.968069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:02.968176 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:02.968563 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:02.968566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:02.969024 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:02.969205 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.969212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.970269 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.970273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.971207 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:02.971218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.971885 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.971889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:02.972005 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.972602 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:02.972619 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:02.975257 (+ 2638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":272,"threads_started":1}
W20260620 15:01:02.975706 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:02.967600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:44615 (local address 127.2.70.62:35577)
0620 15:01:02.967774 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:02.967778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:02.968145 (+ 367us) server_negotiation.cc:407] Connection header received
0620 15:01:02.968334 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:02.968338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:02.968411 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:02.968498 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:02.969334 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.970139 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.971325 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:02.971736 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:02.971829 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:02.975458 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:44615: BlockingRecv error: recv got EOF from 127.2.70.1:44615 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 15:01:03 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:03 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:03.429952 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:03.420944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:59809)
0620 15:01:03.421565 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:03.421583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:03.421761 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:03.422128 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:03.422132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:03.422698 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:03.422896 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:03.422905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.423939 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.423946 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:03.424905 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:03.424916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.425697 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.425704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:03.425869 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:03.426484 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:03.426502 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:03.429720 (+ 3218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":298,"threads_started":1}
W20260620 15:01:03.430154 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:03.421061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:59809 (local address 127.2.70.62:35577)
0620 15:01:03.421306 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:03.421313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:03.421679 (+ 366us) server_negotiation.cc:407] Connection header received
0620 15:01:03.421918 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:03.421922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:03.421983 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:03.422094 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:03.423042 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.423794 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.425057 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.425469 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.425651 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:03.429920 (+ 4269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:59809: BlockingRecv error: recv got EOF from 127.2.70.2:59809 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 15:01:03 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:03 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:03.899180 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:03.890458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:56771)
0620 15:01:03.890940 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:03.890960 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:03.891071 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:03.891404 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:03.891408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:03.891887 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:03.892092 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:03.892101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.893068 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.893072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:03.894127 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:03.894138 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.894848 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.894856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:03.895041 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:03.895768 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:03.895785 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:03.898860 (+ 3075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
W20260620 15:01:03.899282 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:03.890674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:56771 (local address 127.2.70.62:35577)
0620 15:01:03.890881 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:03.890887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:03.891050 (+ 163us) server_negotiation.cc:407] Connection header received
0620 15:01:03.891174 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:03.891177 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:03.891235 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:03.891314 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:03.892221 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.892925 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.894250 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.894635 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.894726 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:03.899041 (+ 4315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:56771: BlockingRecv error: recv got EOF from 127.2.70.3:56771 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 15:01:03 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:03 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:03.985445 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:03.976739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:54829)
0620 15:01:03.977380 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:03.977397 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:03.977618 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:03.977993 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:03.977997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:03.978436 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:03.978577 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:03.978586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.979621 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.979633 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:03.980660 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:03.980673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.981544 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.981552 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:03.981734 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:03.982346 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:03.982363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:03.985123 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":343,"threads_started":1}
W20260620 15:01:03.985636 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:03.976811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:54829 (local address 127.2.70.62:35577)
0620 15:01:03.977120 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:03.977129 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:03.977551 (+ 422us) server_negotiation.cc:407] Connection header received
0620 15:01:03.977778 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:03.977782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:03.977842 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:03.977958 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:03.978720 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.979477 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.980795 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:03.981294 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:03.981441 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:03.985334 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:54829: BlockingRecv error: recv got EOF from 127.2.70.1:54829 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Jun 20 15:01:04 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:04 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:04.442123 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.430955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:56277)
0620 15:01:04.431463 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.431480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.431600 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.431896 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.431921 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.432288 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:04.432432 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.432440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.433610 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.433614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.434519 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.434530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.435195 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.435199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.435303 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.438688 (+ 3385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:04.438719 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:04.441871 (+ 3152us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":286,"threads_started":1}
W20260620 15:01:04.442276 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:04.431010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:56277 (local address 127.2.70.62:35577)
0620 15:01:04.431196 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.431200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.431557 (+ 357us) server_negotiation.cc:407] Connection header received
0620 15:01:04.431702 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.431705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.431765 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.431852 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.432625 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.433367 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.434649 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.435059 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.435167 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.442054 (+ 6887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:56277: BlockingRecv error: recv got EOF from 127.2.70.2:56277 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 15:01:04 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:04 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:04.908492 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.900252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:39973)
0620 15:01:04.900771 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.900791 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.900899 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.901183 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.901186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.901632 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:04.901773 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.901781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.903009 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.903014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.904014 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.904026 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.904738 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.904743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.904881 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.905445 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:04.905463 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:04.908167 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":311,"threads_started":1}
W20260620 15:01:04.908633 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:04.900387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:39973 (local address 127.2.70.62:35577)
0620 15:01:04.900637 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.900643 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.900871 (+ 228us) server_negotiation.cc:407] Connection header received
0620 15:01:04.900996 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.901001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.901060 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.901140 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.901948 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.902792 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.904171 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.904593 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.904686 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.908325 (+ 3639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:39973: BlockingRecv error: recv got EOF from 127.2.70.3:39973 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 15:01:04 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:04 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:04.995102 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:04.986588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:46497)
0620 15:01:04.987072 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:04.987088 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:04.987238 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:04.987569 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:04.987574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:04.987932 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:04.988089 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.988098 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.989087 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.989095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.989997 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:04.990008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.990682 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.990687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:04.990865 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.991563 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:04.991587 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:04.994745 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":250,"threads_started":1}
W20260620 15:01:04.995208 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:04.986650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:46497 (local address 127.2.70.62:35577)
0620 15:01:04.986892 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:04.986901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:04.987173 (+ 272us) server_negotiation.cc:407] Connection header received
0620 15:01:04.987366 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:04.987377 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:04.987433 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:04.987545 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:04.988235 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.988952 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.990134 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:04.990540 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:04.990669 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:04.994963 (+ 4294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:46497: BlockingRecv error: recv got EOF from 127.2.70.1:46497 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 15:01:05 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:05 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:05.451824 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:05.443049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:50475)
0620 15:01:05.443607 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:05.443629 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:05.443762 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:05.444169 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:05.444173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:05.444604 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:05.444842 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:05.444856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.445987 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.445994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:05.446948 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:05.446959 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.447652 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.447658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:05.447863 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:05.448561 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:05.448577 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:05.451591 (+ 3014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":318,"threads_started":1}
W20260620 15:01:05.452176 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:05.443139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:50475 (local address 127.2.70.62:35577)
0620 15:01:05.443370 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:05.443378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:05.443740 (+ 362us) server_negotiation.cc:407] Connection header received
0620 15:01:05.443906 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:05.443913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:05.443988 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:05.444093 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:05.445004 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.445857 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.447091 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.447505 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.447631 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:05.451855 (+ 4224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:50475: BlockingRecv error: recv got EOF from 127.2.70.2:50475 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 15:01:05 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:05 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:05.917960 11561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:05.909580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:56879)
0620 15:01:05.910105 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:05.910136 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:05.910241 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:05.910578 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:05.910581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:05.910908 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:05.911125 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:05.911137 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.912521 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.912544 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:05.913461 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:05.913472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.914157 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.914161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:05.914277 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:05.914786 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:05.914802 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:05.917572 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":325,"threads_started":1}
W20260620 15:01:05.918021 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:05.909687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:56879 (local address 127.2.70.62:35577)
0620 15:01:05.909943 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:05.909949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:05.910207 (+ 258us) server_negotiation.cc:407] Connection header received
0620 15:01:05.910367 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:05.910371 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:05.910434 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:05.910519 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:05.911282 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.912379 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.913641 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.914041 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.914135 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:05.917791 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:56879: BlockingRecv error: recv got EOF from 127.2.70.3:56879 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 15:01:05 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:05 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:06.004223 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:05.996075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:50473)
0620 15:01:05.996567 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:05.996587 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:05.996710 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:05.997018 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:05.997022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:05.997367 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:05.997590 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:05.997608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.998709 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.998713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:05.999672 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:05.999684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.000369 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.000374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:06.000510 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:06.001166 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:06.001184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:06.004004 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":273,"threads_started":1}
W20260620 15:01:06.004532 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:05.996178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:50473 (local address 127.2.70.62:35577)
0620 15:01:05.996462 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:05.996471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:05.996688 (+ 217us) server_negotiation.cc:407] Connection header received
0620 15:01:05.996801 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:05.996805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:05.996868 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:05.996963 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:05.997733 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:05.998579 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:05.999816 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.000245 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.000364 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:06.004223 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:50473: BlockingRecv error: recv got EOF from 127.2.70.1:50473 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 15:01:06 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:06 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:06.460801 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:06.452661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:34453)
0620 15:01:06.453140 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:06.453157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:06.453273 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:06.453725 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:06.453730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:06.454107 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:06.454250 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:06.454258 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.455231 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.455235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:06.456204 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:06.456216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.456941 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.456946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:06.457091 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:06.457784 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:06.457800 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:06.460564 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":275,"threads_started":1}
W20260620 15:01:06.461040 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:06.452865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:34453 (local address 127.2.70.62:35577)
0620 15:01:06.453078 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:06.453086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:06.453232 (+ 146us) server_negotiation.cc:407] Connection header received
0620 15:01:06.453366 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:06.453370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:06.453537 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:06.453642 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:06.454383 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.455114 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.456339 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.456771 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.456863 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:06.460798 (+ 3935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:34453: BlockingRecv error: recv got EOF from 127.2.70.2:34453 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 15:01:06 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:06 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:06.927592 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:06.918931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:38551)
0620 15:01:06.919419 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:06.919483 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:06.919627 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:06.920067 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:06.920074 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:06.920588 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:06.920804 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:06.920819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.922176 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.922180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:06.923111 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:06.923122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.923737 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.923741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:06.923862 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:06.924507 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:06.924523 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:06.927373 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":250,"threads_started":1}
W20260620 15:01:06.927847 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:06.919081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:38551 (local address 127.2.70.62:35577)
0620 15:01:06.919294 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:06.919301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:06.919585 (+ 284us) server_negotiation.cc:407] Connection header received
0620 15:01:06.919754 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:06.919761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:06.919828 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:06.919909 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:06.921006 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.922060 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.923224 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:06.923625 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:06.923711 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:06.927577 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:38551: BlockingRecv error: recv got EOF from 127.2.70.3:38551 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 15:01:07 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:07 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:07.013455 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:07.005227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:36943)
0620 15:01:07.005759 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:07.005774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:07.005930 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:07.006304 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:07.006309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:07.006729 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:07.006888 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:07.006896 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.007907 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.007914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:07.008824 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:07.008834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.009640 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.009646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:07.009818 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:07.010520 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:07.010539 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:07.013223 (+ 2684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":294,"threads_started":1}
W20260620 15:01:07.013707 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:07.005306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:36943 (local address 127.2.70.62:35577)
0620 15:01:07.005577 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:07.005583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:07.005865 (+ 282us) server_negotiation.cc:407] Connection header received
0620 15:01:07.006095 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:07.006100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:07.006174 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:07.006274 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:07.007040 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.007768 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.008963 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.009460 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.009618 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:07.013418 (+ 3800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:36943: BlockingRecv error: recv got EOF from 127.2.70.1:36943 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 15:01:07 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:07 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:07.469942 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:07.461923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:34647)
0620 15:01:07.462427 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:07.462450 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:07.462555 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:07.462823 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:07.462826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:07.463194 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:07.463349 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:07.463356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.464308 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.464312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:07.465379 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:07.465391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.466141 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.466145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:07.466266 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:07.466864 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:07.466880 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:07.469599 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":252,"threads_started":1}
W20260620 15:01:07.470088 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:07.462152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:34647 (local address 127.2.70.62:35577)
0620 15:01:07.462316 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:07.462320 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:07.462533 (+ 213us) server_negotiation.cc:407] Connection header received
0620 15:01:07.462645 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:07.462649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:07.462706 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:07.462783 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:07.463461 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.464178 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.465553 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.465970 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.466061 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:07.469785 (+ 3724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:34647: BlockingRecv error: recv got EOF from 127.2.70.2:34647 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 15:01:07 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:07 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:07.937085 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:07.928588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:52247)
0620 15:01:07.929172 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:07.929205 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:07.929318 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:07.929777 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:07.929781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:07.930325 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:07.930587 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:07.930599 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.931695 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.931700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:07.932640 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:07.932651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.933305 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.933309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:07.933477 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:07.934034 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:07.934051 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:07.936854 (+ 2803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":285,"threads_started":1}
W20260620 15:01:07.937378 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:07.928633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:52247 (local address 127.2.70.62:35577)
0620 15:01:07.928862 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:07.928872 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:07.929302 (+ 430us) server_negotiation.cc:407] Connection header received
0620 15:01:07.929468 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:07.929473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:07.929624 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:07.929739 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:07.930738 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.931542 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.932776 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:07.933186 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:07.933277 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:07.937100 (+ 3823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:52247: BlockingRecv error: recv got EOF from 127.2.70.3:52247 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 15:01:08 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:08 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:08.023136 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:08.014567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:55903)
0620 15:01:08.015123 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:08.015138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:08.015247 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:08.015567 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:08.015570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:08.015924 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:08.016077 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:08.016085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.017170 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.017175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:08.018347 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:08.018361 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.019196 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.019200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:08.019324 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:08.020025 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:08.020043 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:08.022916 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":299,"threads_started":1}
W20260620 15:01:08.023509 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:08.014619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:55903 (local address 127.2.70.62:35577)
0620 15:01:08.014819 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:08.014824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:08.015218 (+ 394us) server_negotiation.cc:407] Connection header received
0620 15:01:08.015346 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:08.015350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:08.015409 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:08.015502 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:08.016207 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.016987 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.018584 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.019044 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.019154 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:08.023161 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:55903: BlockingRecv error: recv got EOF from 127.2.70.1:55903 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 15:01:08 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:08 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:08.480777 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:08.470990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:36413)
0620 15:01:08.471674 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:08.471704 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:08.471922 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:08.472312 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:08.472317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:08.472768 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:08.472939 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:08.472948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.474418 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.474426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:08.475792 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:08.475804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.476555 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.476560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:08.476673 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:08.477318 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:08.477335 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:08.480321 (+ 2986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":314,"threads_started":1}
W20260620 15:01:08.480885 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:08.471144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:36413 (local address 127.2.70.62:35577)
0620 15:01:08.471388 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:08.471396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:08.471827 (+ 431us) server_negotiation.cc:407] Connection header received
0620 15:01:08.472073 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:08.472079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:08.472160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:08.472278 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:08.473119 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.474226 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.475957 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.476380 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.476503 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:08.480620 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:36413: BlockingRecv error: recv got EOF from 127.2.70.2:36413 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 15:01:08 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:08 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:08.945801 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:08.938135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:41509)
0620 15:01:08.938693 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:08.938713 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:08.938821 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:08.939106 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:08.939110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:08.939468 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:08.939611 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:08.939618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.940563 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.940567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:08.941467 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:08.941534 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.942207 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.942211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:08.942330 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:08.942861 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:08.942877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:08.945541 (+ 2664us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":316,"threads_started":1}
W20260620 15:01:08.946082 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:08.938231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:41509 (local address 127.2.70.62:35577)
0620 15:01:08.938419 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:08.938425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:08.938789 (+ 364us) server_negotiation.cc:407] Connection header received
0620 15:01:08.938916 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:08.938919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:08.938976 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:08.939060 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:08.939725 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.940439 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.941661 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:08.942083 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:08.942173 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:08.945813 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:41509: BlockingRecv error: recv got EOF from 127.2.70.3:41509 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 15:01:09 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:09 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:09.032927 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:09.024135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:47571)
0620 15:01:09.024703 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:09.024719 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:09.024836 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:09.025183 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:09.025187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:09.025629 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:09.025793 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:09.025802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.026805 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.026809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:09.027779 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:09.027790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.028488 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.028492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:09.028595 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:09.029304 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:09.029327 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:09.032580 (+ 3253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":280,"threads_started":1}
W20260620 15:01:09.032990 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:09.024276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:47571 (local address 127.2.70.62:35577)
0620 15:01:09.024560 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:09.024566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:09.024801 (+ 235us) server_negotiation.cc:407] Connection header received
0620 15:01:09.024965 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:09.024969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:09.025032 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:09.025112 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:09.025918 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.026653 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.027930 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.028362 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.028455 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:09.032769 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:47571: BlockingRecv error: recv got EOF from 127.2.70.1:47571 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Jun 20 15:01:09 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:09 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:09.494204 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:09.482030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:40273)
0620 15:01:09.482720 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:09.482737 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:09.482855 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:09.483222 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:09.483226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:09.483613 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:09.483785 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:09.483793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.484767 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.484771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:09.485805 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:09.485820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.486479 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.486483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:09.486619 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:09.491011 (+ 4392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:09.491033 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:09.493941 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":356,"threads_started":1}
W20260620 15:01:09.494467 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:09.482066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:40273 (local address 127.2.70.62:35577)
0620 15:01:09.482294 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:09.482299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:09.482818 (+ 519us) server_negotiation.cc:407] Connection header received
0620 15:01:09.482957 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:09.482961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:09.483061 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:09.483150 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:09.483906 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.484637 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.485944 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.486344 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.486444 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:09.494140 (+ 7696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:40273: BlockingRecv error: recv got EOF from 127.2.70.2:40273 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Jun 20 15:01:09 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:09 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:09.955693 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:09.946969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:54407)
0620 15:01:09.947621 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:09.947645 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:09.947805 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:09.948228 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:09.948232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:09.948981 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:09.949165 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:09.949175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.950303 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.950307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:09.951247 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:09.951258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.951921 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.951926 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:09.952033 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:09.952685 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:09.952703 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:09.955463 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":247,"threads_started":1}
W20260620 15:01:09.955770 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:09.947112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:54407 (local address 127.2.70.62:35577)
0620 15:01:09.947323 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:09.947331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:09.947739 (+ 408us) server_negotiation.cc:407] Connection header received
0620 15:01:09.947940 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:09.947945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:09.948017 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:09.948159 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:09.949328 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.950151 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.951378 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:09.951782 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:09.951876 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:09.955607 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:54407: BlockingRecv error: recv got EOF from 127.2.70.3:54407 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 15:01:10 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:10 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:10.042865 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:10.033927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:35165)
0620 15:01:10.034457 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:10.034472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:10.034653 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:10.034996 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:10.035001 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:10.035420 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:10.035626 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:10.035645 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.037102 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.037109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:10.038103 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:10.038115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.038969 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.038978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:10.039145 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:10.039794 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:10.039812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:10.042544 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":299,"threads_started":1}
W20260620 15:01:10.042943 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:10.034082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:35165 (local address 127.2.70.62:35577)
0620 15:01:10.034310 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:10.034317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:10.034585 (+ 268us) server_negotiation.cc:407] Connection header received
0620 15:01:10.034783 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:10.034791 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:10.034854 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:10.034957 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:10.035787 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.036956 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.038243 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.038779 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.038909 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:10.042702 (+ 3793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:35165: BlockingRecv error: recv got EOF from 127.2.70.1:35165 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 15:01:10 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:10 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:10.503839 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:10.495245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:58465)
0620 15:01:10.495730 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:10.495747 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:10.495873 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:10.496179 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:10.496183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:10.496603 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:10.496751 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:10.496759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.497969 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.497973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:10.499074 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:10.499087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.499951 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.499958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:10.500126 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:10.500881 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:10.500899 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:10.503619 (+ 2720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":283,"threads_started":1}
W20260620 15:01:10.504015 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:10.495302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:58465 (local address 127.2.70.62:35577)
0620 15:01:10.495515 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:10.495524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:10.495808 (+ 284us) server_negotiation.cc:407] Connection header received
0620 15:01:10.495994 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:10.495999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:10.496063 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:10.496139 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:10.496873 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.497834 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.499237 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.499775 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.499866 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:10.503783 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:58465: BlockingRecv error: recv got EOF from 127.2.70.2:58465 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 15:01:10 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:10 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:10.965723 11576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:10.956793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:42151)
0620 15:01:10.957416 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:10.957445 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:10.957614 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:10.957987 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:10.957991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:10.958547 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:10.958754 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:10.958768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.959953 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.959957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:10.960925 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:10.960942 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.961753 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.961760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:10.961942 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:10.962546 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:10.962563 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:10.965338 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":356,"threads_started":1}
W20260620 15:01:10.965914 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:10.956899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:42151 (local address 127.2.70.62:35577)
0620 15:01:10.957211 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:10.957217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:10.957604 (+ 387us) server_negotiation.cc:407] Connection header received
0620 15:01:10.957747 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:10.957755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:10.957825 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:10.957909 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:10.958948 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.959808 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.961091 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:10.961580 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:10.961719 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:10.965668 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:42151: BlockingRecv error: recv got EOF from 127.2.70.3:42151 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 20 15:01:11 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:11 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:11.052160 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:11.043965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:42755)
0620 15:01:11.044448 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:11.044464 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:11.044619 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:11.044962 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:11.044967 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:11.045368 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:11.045558 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:11.045568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.046576 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.046583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:11.047508 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:11.047519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.048248 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.048254 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:11.048429 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:11.049053 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:11.049071 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:11.051795 (+ 2724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":237,"threads_started":1}
W20260620 15:01:11.052201 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:11.043989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:42755 (local address 127.2.70.62:35577)
0620 15:01:11.044216 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:11.044224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:11.044553 (+ 329us) server_negotiation.cc:407] Connection header received
0620 15:01:11.044749 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:11.044754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:11.044829 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:11.044928 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:11.045701 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.046433 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.047645 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.048097 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.048231 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:11.051983 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:42755: BlockingRecv error: recv got EOF from 127.2.70.1:42755 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 15:01:11 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:11 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 15:01:11.515890 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:11.504833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:34039 (local address 127.2.70.62:35577)
0620 15:01:11.505055 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:11.505063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:11.505383 (+ 320us) server_negotiation.cc:407] Connection header received
0620 15:01:11.505603 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:11.505608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:11.505676 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:11.505776 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:11.506619 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.507738 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.509420 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.510136 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.510230 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:11.515702 (+ 5472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:34039: BlockingRecv error: recv got EOF from 127.2.70.2:34039 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:01:11.515938 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:11.504747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:34039)
0620 15:01:11.505281 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:11.505297 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:11.505449 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:11.505808 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:11.505812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:11.506260 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:11.506483 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:11.506493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.507876 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.507883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:11.509245 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:11.509263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.510507 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.510513 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:11.510690 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:11.511377 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:11.511402 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:11.515522 (+ 4120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":309,"threads_started":1}
Jun 20 15:01:11 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:11 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:11.976006 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:11.966651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:53111)
0620 15:01:11.967229 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:11.967250 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:11.967485 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:11.967861 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:11.967866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:11.968325 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:11.968515 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:11.968525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.969716 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.969723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:11.970767 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:11.970778 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.971571 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.971578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:11.971768 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:11.972521 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:11.972547 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:11.975621 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":319,"threads_started":1}
W20260620 15:01:11.976164 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:11.966739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:53111 (local address 127.2.70.62:35577)
0620 15:01:11.967018 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:11.967025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:11.967349 (+ 324us) server_negotiation.cc:407] Connection header received
0620 15:01:11.967631 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:11.967637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:11.967714 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:11.967829 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:11.968679 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.969568 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.970931 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:11.971398 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:11.971545 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:11.975903 (+ 4358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:53111: BlockingRecv error: recv got EOF from 127.2.70.3:53111 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 15:01:12 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:12 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:12.061435 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:12.053048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:34495)
0620 15:01:12.053605 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:12.053634 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:12.053753 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:12.054049 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:12.054053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:12.054440 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:12.054587 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:12.054596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.055583 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.055588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:12.056648 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:12.056661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.057334 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.057339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:12.057514 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:12.058082 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:12.058104 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:12.061104 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":343,"threads_started":1}
W20260620 15:01:12.061688 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:12.053185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:34495 (local address 127.2.70.62:35577)
0620 15:01:12.053357 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:12.053362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:12.053720 (+ 358us) server_negotiation.cc:407] Connection header received
0620 15:01:12.053859 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:12.053862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:12.053920 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:12.054013 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:12.054714 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.055450 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.056773 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.057191 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.057278 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:12.061321 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:34495: BlockingRecv error: recv got EOF from 127.2.70.1:34495 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 15:01:12 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:12 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:12.525051 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:12.517107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:49847)
0620 15:01:12.517659 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:12.517676 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:12.517837 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:12.518182 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:12.518187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:12.518647 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:12.518787 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:12.518795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.519838 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.519844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:12.520751 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:12.520761 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.521456 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.521462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:12.521665 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:12.522186 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:12.522203 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:12.524832 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":275,"threads_started":1}
W20260620 15:01:12.525241 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:12.517180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:49847 (local address 127.2.70.62:35577)
0620 15:01:12.517452 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:12.517460 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:12.517765 (+ 305us) server_negotiation.cc:407] Connection header received
0620 15:01:12.517965 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:12.517970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:12.518051 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:12.518154 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:12.518928 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.519701 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.520888 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.521296 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.521444 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:12.525026 (+ 3582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:49847: BlockingRecv error: recv got EOF from 127.2.70.2:49847 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 20 15:01:12 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:12 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:12.985610 11582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:12.977125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:53877)
0620 15:01:12.977734 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:12.977759 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:12.977910 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:12.978268 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:12.978272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:12.978655 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:12.978792 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:12.978800 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.979812 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.979818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:12.980743 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:12.980753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.981474 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.981516 (+ 42us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:12.981700 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:12.982273 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:12.982289 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:12.985281 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":353,"threads_started":1}
W20260620 15:01:12.985767 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:12.977278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:53877 (local address 127.2.70.62:35577)
0620 15:01:12.977546 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:12.977552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:12.977849 (+ 297us) server_negotiation.cc:407] Connection header received
0620 15:01:12.978048 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:12.978053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:12.978145 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:12.978245 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:12.978930 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.979674 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.980881 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:12.981308 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:12.981462 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:12.985468 (+ 4006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:53877: BlockingRecv error: recv got EOF from 127.2.70.3:53877 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 15:01:13 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:13 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:13.070518 11583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:13.062385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:43733)
0620 15:01:13.062823 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:13.062859 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:13.062969 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:13.063237 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:13.063239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:13.063639 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:13.063802 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:13.063811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.064770 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.064774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:13.065891 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:13.065904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.066631 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.066636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:13.066750 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:13.067438 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:13.067456 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:13.070282 (+ 2826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
W20260620 15:01:13.070889 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:13.062425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:43733 (local address 127.2.70.62:35577)
0620 15:01:13.062661 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:13.062668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:13.062936 (+ 268us) server_negotiation.cc:407] Connection header received
0620 15:01:13.063072 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:13.063076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:13.063147 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:13.063224 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:13.063923 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.064637 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.066044 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.066477 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.066580 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:13.070480 (+ 3900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:43733: BlockingRecv error: recv got EOF from 127.2.70.1:43733 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 15:01:13 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:13 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:13.534626 11584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:13.526054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:52643)
0620 15:01:13.526525 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:13.526541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:13.526661 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:13.526951 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:13.526955 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:13.527443 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:13.527629 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:13.527638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.528661 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.528666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:13.530014 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:13.530027 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.530833 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.530838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:13.530956 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:13.531595 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:13.531612 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:13.534383 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
W20260620 15:01:13.534819 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:13.526135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:52643 (local address 127.2.70.62:35577)
0620 15:01:13.526354 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:13.526360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:13.526620 (+ 260us) server_negotiation.cc:407] Connection header received
0620 15:01:13.526744 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:13.526747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:13.526805 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:13.526882 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:13.527754 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.528500 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.530165 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.530690 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.530789 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:13.534591 (+ 3802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:52643: BlockingRecv error: recv got EOF from 127.2.70.2:52643 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 15:01:13 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:13 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:13.994848 11585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:13.986564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:58029)
0620 15:01:13.987134 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:13.987160 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:13.987294 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:13.987664 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:13.987669 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:13.988178 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:13.988422 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:13.988436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.989679 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.989683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:13.990576 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:13.990587 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.991204 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.991208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:13.991315 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:13.991910 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:13.991927 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:13.994615 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":455,"threads_started":1}
W20260620 15:01:13.995082 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:13.986845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:58029 (local address 127.2.70.62:35577)
0620 15:01:13.987005 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:13.987011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:13.987262 (+ 251us) server_negotiation.cc:407] Connection header received
0620 15:01:13.987403 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:13.987410 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:13.987475 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:13.987594 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:13.988569 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.989562 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.990694 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:13.991081 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:13.991175 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:13.994815 (+ 3640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:58029: BlockingRecv error: recv got EOF from 127.2.70.3:58029 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 15:01:14 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:14 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:14.079849 11586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:14.071471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:41029)
0620 15:01:14.072014 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:14.072030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:14.072188 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:14.072551 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:14.072555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:14.072961 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:14.073139 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:14.073147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.074214 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.074220 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:14.075125 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:14.075136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.075920 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.075925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:14.076114 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:14.076718 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:14.076734 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:14.079613 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":283,"threads_started":1}
W20260620 15:01:14.080060 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:14.071530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:41029 (local address 127.2.70.62:35577)
0620 15:01:14.071775 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:14.071784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:14.072117 (+ 333us) server_negotiation.cc:407] Connection header received
0620 15:01:14.072329 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:14.072335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:14.072403 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:14.072538 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:14.073282 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.074076 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.075275 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.075769 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.075896 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:14.079804 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:41029: BlockingRecv error: recv got EOF from 127.2.70.1:41029 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 20 15:01:14 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:14 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:14.547124 11587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:14.535548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:44733)
0620 15:01:14.536111 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:14.536127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:14.536236 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:14.536520 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:14.536525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:14.536915 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:14.537084 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:14.537091 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.538130 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.538134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:14.539030 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:14.539041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.539727 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.539732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:14.539873 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:14.543417 (+ 3544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:14.543437 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:14.546864 (+ 3427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":309,"threads_started":1}
W20260620 15:01:14.547242 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:14.535603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:44733 (local address 127.2.70.62:35577)
0620 15:01:14.535786 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:14.535790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:14.536200 (+ 410us) server_negotiation.cc:407] Connection header received
0620 15:01:14.536331 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:14.536335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:14.536393 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:14.536471 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:14.537201 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.537994 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.539157 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.539555 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.539648 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:14.547050 (+ 7402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:44733: BlockingRecv error: recv got EOF from 127.2.70.2:44733 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 15:01:14 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:14 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:15.004096 11588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:14.995820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:48103)
0620 15:01:14.996328 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:14.996347 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:14.996513 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:14.996801 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:14.996804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:14.997173 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:14.997322 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:14.997330 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.998464 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.998472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:14.999608 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:14.999620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.000319 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.000324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:15.000505 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:15.001155 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:15.001173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:15.003848 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":307,"threads_started":1}
W20260620 15:01:15.004276 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:14.995896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:48103 (local address 127.2.70.62:35577)
0620 15:01:14.996150 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:14.996156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:14.996458 (+ 302us) server_negotiation.cc:407] Connection header received
0620 15:01:14.996619 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:14.996623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:14.996684 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:14.996765 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:14.997441 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:14.998310 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:14.999743 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.000169 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.000296 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:15.004047 (+ 3751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:48103: BlockingRecv error: recv got EOF from 127.2.70.3:48103 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 15:01:15 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:15 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:15.089900 11589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:15.080730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:48177)
0620 15:01:15.081313 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:15.081331 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:15.081553 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:15.081958 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:15.081962 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:15.082456 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:15.082709 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:15.082722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.084099 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.084103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:15.085045 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:15.085056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.085813 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.085818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:15.085958 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:15.086751 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:15.086769 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:15.089677 (+ 2908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":293,"threads_started":1}
W20260620 15:01:15.090265 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:15.080928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:48177 (local address 127.2.70.62:35577)
0620 15:01:15.081199 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:15.081207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:15.081455 (+ 248us) server_negotiation.cc:407] Connection header received
0620 15:01:15.081690 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:15.081695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:15.081769 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:15.081884 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:15.082862 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.083967 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.085176 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.085628 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.085724 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:15.089856 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:48177: BlockingRecv error: recv got EOF from 127.2.70.1:48177 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 15:01:15 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:15 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:15.556325 11590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:15.547977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:40361)
0620 15:01:15.548461 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:15.548478 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:15.548597 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:15.548959 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:15.548963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:15.549379 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:15.549623 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:15.549633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.550900 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.550904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:15.551807 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:15.551817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.552472 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.552476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:15.552586 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:15.553149 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:15.553165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:15.556107 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":215,"threads_started":1}
W20260620 15:01:15.556536 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:15.548201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:40361 (local address 127.2.70.62:35577)
0620 15:01:15.548421 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:15.548444 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:15.548565 (+ 121us) server_negotiation.cc:407] Connection header received
0620 15:01:15.548712 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:15.548717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:15.548800 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:15.548877 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:15.549765 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.550780 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.551929 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:15.552324 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:15.552412 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:15.556288 (+ 3876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:40361: BlockingRecv error: recv got EOF from 127.2.70.2:40361 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 20 15:01:16 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:16 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:16.015937 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:16.005198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:34867)
0620 15:01:16.005766 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:16.005785 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:16.005949 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:16.006290 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:16.006297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:16.006781 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:16.007012 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:16.007024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.008339 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.008348 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:16.010375 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:16.010396 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.011276 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.011285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:16.011515 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:16.012257 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:16.012288 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:16.015714 (+ 3426us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":327,"threads_started":1}
W20260620 15:01:16.016314 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:16.005264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:34867 (local address 127.2.70.62:35577)
0620 15:01:16.005467 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:16.005475 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:16.005878 (+ 403us) server_negotiation.cc:407] Connection header received
0620 15:01:16.006086 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:16.006091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:16.006160 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:16.006259 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:16.007156 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.008148 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.010556 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.011077 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.011178 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:16.016023 (+ 4845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:34867: BlockingRecv error: recv got EOF from 127.2.70.3:34867 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 15:01:16 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:16 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:16.102589 11592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:16.090801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:41567)
0620 15:01:16.091515 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:16.091531 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:16.091676 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:16.092132 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:16.092135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:16.092617 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:16.092868 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:16.092880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.094357 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.094367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:16.096308 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:16.096327 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.097219 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.097228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:16.097519 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:16.098334 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:16.098363 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:16.102242 (+ 3879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":468,"threads_started":1}
W20260620 15:01:16.102815 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:16.090928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:41567 (local address 127.2.70.62:35577)
0620 15:01:16.091325 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:16.091333 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:16.091619 (+ 286us) server_negotiation.cc:407] Connection header received
0620 15:01:16.091857 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:16.091865 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:16.091968 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:16.092095 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:16.093035 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.094156 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.096534 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.097040 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.097165 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:16.102503 (+ 5338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:41567: BlockingRecv error: recv got EOF from 127.2.70.1:41567 (error 108)
Metrics: {"server-negotiator.queue_time_us":201}
Jun 20 15:01:16 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:16 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:16.565209 11593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:16.557219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:57555)
0620 15:01:16.557753 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:16.557770 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:16.557880 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:16.558204 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:16.558208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:16.558580 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:16.558725 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:16.558733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.559707 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.559711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:16.560638 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:16.560649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.561288 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.561292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:16.561416 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:16.562124 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:16.562141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:16.564828 (+ 2687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":314,"threads_started":1}
W20260620 15:01:16.565220 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:16.557344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:57555 (local address 127.2.70.62:35577)
0620 15:01:16.557568 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:16.557572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:16.557843 (+ 271us) server_negotiation.cc:407] Connection header received
0620 15:01:16.557976 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:16.557980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:16.558037 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:16.558121 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:16.558856 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.559586 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.560755 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:16.561170 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:16.561257 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:16.565009 (+ 3752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:57555: BlockingRecv error: recv got EOF from 127.2.70.2:57555 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 15:01:17 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:17 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.3@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:17.024817 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:17.017070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.3:35247)
0620 15:01:17.017631 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:17.017660 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:17.017766 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:17.018103 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:17.018106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:17.018447 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:17.018607 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:17.018614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.019572 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.019576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:17.020480 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:17.020490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.021113 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.021117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:17.021232 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.021936 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:17.021954 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:17.024591 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":322,"threads_started":1}
W20260620 15:01:17.025030 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:17.017160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:35247 (local address 127.2.70.62:35577)
0620 15:01:17.017345 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:17.017351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:17.017728 (+ 377us) server_negotiation.cc:407] Connection header received
0620 15:01:17.017867 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:17.017871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:17.017957 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:17.018060 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:17.018727 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.019445 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.020597 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.020990 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.021085 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.024798 (+ 3713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.3:35247: BlockingRecv error: recv got EOF from 127.2.70.3:35247 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 15:01:17 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:17 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.1@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:17.113379 11595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:17.103967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.1:56275)
0620 15:01:17.104473 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:17.104489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:17.104644 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:17.104968 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:17.104973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:17.105399 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:17.105629 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:17.105640 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.106791 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.106798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:17.108203 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:17.108221 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.109166 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.109173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:17.109350 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.110090 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:17.110108 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:17.113125 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":265,"threads_started":1}
W20260620 15:01:17.113444 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:17.104050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:56275 (local address 127.2.70.62:35577)
0620 15:01:17.104286 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:17.104295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:17.104576 (+ 281us) server_negotiation.cc:407] Connection header received
0620 15:01:17.104784 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:17.104788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:17.104843 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:17.104940 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:17.105776 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.106632 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.108397 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.108993 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.109148 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.113265 (+ 4117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.1:56275: BlockingRecv error: recv got EOF from 127.2.70.1:56275 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 15:01:17 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 15:01:17 dist-test-slave-n5vv krb5kdc[10551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.70.2@KRBTEST.COM for oryx/127.2.70.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 15:01:17.574586 11596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:17.566474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35577 (local address 127.2.70.2:36377)
0620 15:01:17.567028 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:17.567045 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:17.567153 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:17.567419 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:17.567422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:17.567805 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:17.567964 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:17.567971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.568937 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.568941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:17.570054 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:17.570066 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.570782 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.570788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:17.570957 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.571611 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:17.571628 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:17.574327 (+ 2699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.70.62:35577: Server oryx/127.2.70.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":303,"threads_started":1}
W20260620 15:01:17.574726 11287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:17.566533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:36377 (local address 127.2.70.62:35577)
0620 15:01:17.566700 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:17.566705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:17.567118 (+ 413us) server_negotiation.cc:407] Connection header received
0620 15:01:17.567234 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:17.567238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:17.567294 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:17.567377 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:17.568111 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.568816 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.570155 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.570628 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.570777 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.574508 (+ 3731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.70.2:36377: BlockingRecv error: recv got EOF from 127.2.70.2:36377 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:01:17.861361 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11138
I20260620 15:01:17.873961 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:17.862594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.2.70.1:41723)
0620 15:01:17.863192 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:17.863199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:17.863225 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:01:17.863285 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:17.863292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:17.863545 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:17.863704 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:17.865271 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.866523 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.867718 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.868261 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.868360 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.870937 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:17.870946 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:17.870960 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:17.870996 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:17.873403 (+ 2407us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.70.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:01:17.873728 (+ 325us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40800: Request ticket server kudu/127.2.70.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":298,"threads_started":1}
W20260620 15:01:17.874086 11598 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40800: Request ticket server kudu/127.2.70.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:01:17.874142 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11138: Not authorized: Client connection negotiation failed: client connection to 127.2.70.1:41723: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.70.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:01:17.874377 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 11138
I20260620 15:01:17.891625 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11289
I20260620 15:01:17.903934 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:17.892580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.2.70.2:37339)
0620 15:01:17.893214 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:17.893221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:17.893253 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:01:17.893314 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:17.893320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:17.893602 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:17.893763 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:17.895159 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.896310 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.897463 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.898084 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.898179 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.901066 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:17.901074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:17.901088 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:17.901122 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:17.903436 (+ 2314us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.70.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:01:17.903705 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33988: Request ticket server kudu/127.2.70.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":327,"threads_started":1}
W20260620 15:01:17.904052 11599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33988: Request ticket server kudu/127.2.70.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:01:17.904085 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11289: Not authorized: Client connection negotiation failed: client connection to 127.2.70.2:37339: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.70.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:01:17.904239 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 11289
I20260620 15:01:17.920960 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11420
I20260620 15:01:17.932381 11600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:17.921866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.2.70.3:42773)
0620 15:01:17.922494 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:17.922504 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:17.922534 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:01:17.922620 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:17.922627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:17.922853 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:17.923017 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:17.924203 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.925208 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.926478 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:17.927037 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:17.927148 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:17.929600 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:17.929610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:17.929626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:17.929659 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:17.931876 (+ 2217us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.70.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0620 15:01:17.932149 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41844: Request ticket server kudu/127.2.70.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":359,"threads_started":1}
W20260620 15:01:17.932503 11600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41844: Request ticket server kudu/127.2.70.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260620 15:01:17.932557 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11420: Not authorized: Client connection negotiation failed: client connection to 127.2.70.3:42773: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.2.70.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260620 15:01:17.932695 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 11420
I20260620 15:01:17.948832 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11069
I20260620 15:01:17.950484 11131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:18.065387 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 11069
2026-06-20T15:01:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21099 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T15:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:18Z Disabled control of system clock
I20260620 15:01:18.141306 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:41095
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:41095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 15:01:18.469450 11605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:18.469955 11605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:18.470110 11605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:18.477777 11605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:18.477902 11605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:18.477942 11605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:18.477999 11605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:18.478042 11605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:18.487298 11605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:41095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:41095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:18.488598 11605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:18.490314 11605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:18.502272 11613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:18.502324 11611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:18.502226 11610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:18.507179 11605 server_base.cc:1034] running on GCE node
I20260620 15:01:18.508055 11605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:18.509632 11605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:18.510849 11605 hybrid_clock.cc:648] HybridClock initialized: now 1781967678510792 us; error 52 us; skew 500 ppm
I20260620 15:01:18.511286 11605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:01:18.514607 11605 webserver.cc:460] Webserver started at http://127.2.70.62:38403/ using document root <none> and password file <none>
I20260620 15:01:18.515347 11605 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:18.515461 11605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:18.515784 11605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:18.518786 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "62d0392af39a490da8523fec817e4080"
format_stamp: "Formatted at 2026-06-20 15:01:18 on dist-test-slave-n5vv"
server_key: "999335aca2a58c8d08186d2a1afb26c4"
server_key_iv: "7a33c45aa5d3a5c129659b1fef43674a"
server_key_version: "encryptionkey@0"
I20260620 15:01:18.519565 11605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "62d0392af39a490da8523fec817e4080"
format_stamp: "Formatted at 2026-06-20 15:01:18 on dist-test-slave-n5vv"
server_key: "999335aca2a58c8d08186d2a1afb26c4"
server_key_iv: "7a33c45aa5d3a5c129659b1fef43674a"
server_key_version: "encryptionkey@0"
I20260620 15:01:18.524930 11605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:01:18.528551 11619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:18.530124 11605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 15:01:18.530306 11605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "62d0392af39a490da8523fec817e4080"
format_stamp: "Formatted at 2026-06-20 15:01:18 on dist-test-slave-n5vv"
server_key: "999335aca2a58c8d08186d2a1afb26c4"
server_key_iv: "7a33c45aa5d3a5c129659b1fef43674a"
server_key_version: "encryptionkey@0"
I20260620 15:01:18.530541 11605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:18.547117 11605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:18.547951 11605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:18.548244 11605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:18.566560 11605 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:41095
I20260620 15:01:18.566577 11670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:41095 every 8 connection(s)
I20260620 15:01:18.567886 11605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:18.571557 11671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:18.572075 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 11605
I20260620 15:01:18.572280 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:01:18.572898 2328 external_mini_cluster.cc:1428] Setting key b3b91f86888fa6a72232470030d10cee
I20260620 15:01:18.584887 11671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080: Bootstrap starting.
I20260620 15:01:18.587200 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:18.574916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.2.70.62:41095)
0620 15:01:18.575892 (+ 976us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:18.575905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:18.575949 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:01:18.577640 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:18.577662 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:18.578253 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:18.578657 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:18.579219 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:18.580444 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:18.582459 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:18.583181 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:18.583318 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:18.584580 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:18.584588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:18.584602 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:18.585223 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:18.585302 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:18.585323 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:18.585416 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:18.585688 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 15:01:18.586050 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":327,"threads_started":1}
I20260620 15:01:18.588338 11671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:18.589387 11671 log.cc:824] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:18.592192 11671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080: No bootstrap required, opened a new log
I20260620 15:01:18.599357 11671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d0392af39a490da8523fec817e4080" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 41095 } }
I20260620 15:01:18.599761 11671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:18.599869 11671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62d0392af39a490da8523fec817e4080, State: Initialized, Role: FOLLOWER
I20260620 15:01:18.600585 11671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d0392af39a490da8523fec817e4080" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 41095 } }
I20260620 15:01:18.600783 11671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:18.600879 11671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:18.601014 11671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:18.603157 11671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d0392af39a490da8523fec817e4080" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 41095 } }
I20260620 15:01:18.603596 11671 leader_election.cc:302] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62d0392af39a490da8523fec817e4080; no voters:
I20260620 15:01:18.604130 11671 leader_election.cc:288] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:18.604295 11676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:18.605067 11676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [term 1 LEADER]: Becoming Leader. State: Replica: 62d0392af39a490da8523fec817e4080, State: Running, Role: LEADER
I20260620 15:01:18.605684 11676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d0392af39a490da8523fec817e4080" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 41095 } }
I20260620 15:01:18.606280 11671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:18.608518 11677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62d0392af39a490da8523fec817e4080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d0392af39a490da8523fec817e4080" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 41095 } } }
I20260620 15:01:18.608963 11677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:18.609923 11685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:18.609833 11678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62d0392af39a490da8523fec817e4080. Latest consensus state: current_term: 1 leader_uuid: "62d0392af39a490da8523fec817e4080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62d0392af39a490da8523fec817e4080" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 41095 } } }
I20260620 15:01:18.610682 11678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:18.615044 11685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:18.623135 11685 catalog_manager.cc:1347] Generated new cluster ID: b765fb88bc8c48d58a5039b207b764a5
I20260620 15:01:18.623256 11685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:18.654117 11685 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:18.655958 11685 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:18.667868 11685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62d0392af39a490da8523fec817e4080: Generated new TSK 0
I20260620 15:01:18.668999 11685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:18.688426 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:41095
--builtin_ntp_servers=127.2.70.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 15:01:19.023298 11695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:19.023654 11695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:19.023825 11695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:19.031440 11695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:19.031600 11695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:19.031785 11695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:01:19.041244 11695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:41095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:19.042652 11695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:19.044474 11695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:19.058727 11701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:19.058969 11703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:19.059240 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:19.059597 11695 server_base.cc:1034] running on GCE node
I20260620 15:01:19.060233 11695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:19.061374 11695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:19.062660 11695 hybrid_clock.cc:648] HybridClock initialized: now 1781967679062590 us; error 116 us; skew 500 ppm
I20260620 15:01:19.063017 11695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:01:19.065284 11695 webserver.cc:460] Webserver started at http://127.2.70.1:34537/ using document root <none> and password file <none>
I20260620 15:01:19.066062 11695 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:19.066174 11695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:19.066519 11695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:19.069334 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "4509e90871474c40ab6732d88169e5de"
format_stamp: "Formatted at 2026-06-20 15:01:19 on dist-test-slave-n5vv"
server_key: "d668d9011cc5395c7fc68c7f8a1af0f6"
server_key_iv: "02cf364126ebbe077eb48d25685868d3"
server_key_version: "encryptionkey@0"
I20260620 15:01:19.070204 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "4509e90871474c40ab6732d88169e5de"
format_stamp: "Formatted at 2026-06-20 15:01:19 on dist-test-slave-n5vv"
server_key: "d668d9011cc5395c7fc68c7f8a1af0f6"
server_key_iv: "02cf364126ebbe077eb48d25685868d3"
server_key_version: "encryptionkey@0"
I20260620 15:01:19.075505 11695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:01:19.079128 11709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:19.080621 11695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 15:01:19.080821 11695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "4509e90871474c40ab6732d88169e5de"
format_stamp: "Formatted at 2026-06-20 15:01:19 on dist-test-slave-n5vv"
server_key: "d668d9011cc5395c7fc68c7f8a1af0f6"
server_key_iv: "02cf364126ebbe077eb48d25685868d3"
server_key_version: "encryptionkey@0"
I20260620 15:01:19.081039 11695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:19.103842 11695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:19.104710 11695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:19.104991 11695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:19.106092 11695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:19.107704 11695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:19.107828 11695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:19.107941 11695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:19.108001 11695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:19.141971 11695 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:42791
I20260620 15:01:19.142066 11821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:42791 every 8 connection(s)
I20260620 15:01:19.143275 11695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:19.153546 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 11695
I20260620 15:01:19.153754 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:19.154372 2328 external_mini_cluster.cc:1428] Setting key fc42f32b36ef137655eca655a030dadc
I20260620 15:01:19.155674 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:19.145903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:41043 (local address 127.2.70.62:41095)
0620 15:01:19.146342 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:19.146347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:19.147385 (+ 1038us) server_negotiation.cc:407] Connection header received
0620 15:01:19.149113 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:19.149118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:19.149207 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:19.149330 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:19.150395 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:19.151137 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:19.152664 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:19.153143 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:19.153306 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:19.154886 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:19.154894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:19.154898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:19.154923 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:19.154967 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:19.154980 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:19.155059 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:19.155314 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 15:01:19.155411 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":246,"threads_started":1}
I20260620 15:01:19.156838 11825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:19.146350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:41095 (local address 127.2.70.1:41043)
0620 15:01:19.147233 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:19.147268 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:19.148855 (+ 1587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:19.149607 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:19.149615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:19.149704 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:19.150234 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:19.150249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:19.151295 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:19.151299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:19.152523 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:19.152534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:19.153410 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:19.153416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:19.153767 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:19.154575 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:19.154578 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:19.154622 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:19.154699 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:19.155090 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:19.155102 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:19.155113 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:01:19.155285 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 15:01:19.155682 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":277,"threads_started":1}
I20260620 15:01:19.157219 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:41095
--builtin_ntp_servers=127.2.70.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 15:01:19.158313 11822 heartbeater.cc:347] Connected to a master server at 127.2.70.62:41095
I20260620 15:01:19.158639 11822 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:19.159521 11822 heartbeater.cc:511] Master 127.2.70.62:41095 requested a full tablet report, sending...
I20260620 15:01:19.161798 11636 ts_manager.cc:194] Registered new tserver with Master: 4509e90871474c40ab6732d88169e5de (127.2.70.1:42791)
I20260620 15:01:19.163965 11636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.1:41043
W20260620 15:01:19.488623 11826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:19.488914 11826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:19.488996 11826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:19.496335 11826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:19.496455 11826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:19.496558 11826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:01:19.505770 11826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:41095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:19.507087 11826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:19.508795 11826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:19.521855 11831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:19.521903 11834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:19.521827 11832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:19.523044 11826 server_base.cc:1034] running on GCE node
I20260620 15:01:19.523674 11826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:19.524628 11826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:19.525861 11826 hybrid_clock.cc:648] HybridClock initialized: now 1781967679525812 us; error 57 us; skew 500 ppm
I20260620 15:01:19.526202 11826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:01:19.528419 11826 webserver.cc:460] Webserver started at http://127.2.70.2:43951/ using document root <none> and password file <none>
I20260620 15:01:19.529129 11826 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:19.529261 11826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:19.529654 11826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:19.532377 11826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "c9683d7ff2f343c8a2777eaf2c94813a"
format_stamp: "Formatted at 2026-06-20 15:01:19 on dist-test-slave-n5vv"
server_key: "1622c6ee6824bf1a1f62b4a096af75cf"
server_key_iv: "5f84393835e5bbf6efd377d983b80551"
server_key_version: "encryptionkey@0"
I20260620 15:01:19.533156 11826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "c9683d7ff2f343c8a2777eaf2c94813a"
format_stamp: "Formatted at 2026-06-20 15:01:19 on dist-test-slave-n5vv"
server_key: "1622c6ee6824bf1a1f62b4a096af75cf"
server_key_iv: "5f84393835e5bbf6efd377d983b80551"
server_key_version: "encryptionkey@0"
I20260620 15:01:19.538450 11826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:01:19.541764 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:19.543021 11826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:01:19.543200 11826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "c9683d7ff2f343c8a2777eaf2c94813a"
format_stamp: "Formatted at 2026-06-20 15:01:19 on dist-test-slave-n5vv"
server_key: "1622c6ee6824bf1a1f62b4a096af75cf"
server_key_iv: "5f84393835e5bbf6efd377d983b80551"
server_key_version: "encryptionkey@0"
I20260620 15:01:19.543416 11826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:19.575392 11826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:19.576229 11826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:19.576516 11826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:19.577462 11826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:19.578991 11826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:19.579114 11826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:19.579241 11826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:19.579326 11826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:19.612097 11826 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:45659
I20260620 15:01:19.612179 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:45659 every 8 connection(s)
I20260620 15:01:19.613351 11826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:19.621707 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 11826
I20260620 15:01:19.621893 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:19.622319 2328 external_mini_cluster.cc:1428] Setting key 3c08ecc4420e953035489e8abc855fe5
I20260620 15:01:19.625392 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:41095
--builtin_ntp_servers=127.2.70.20:33073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 15:01:19.626048 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:19.615855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:58613 (local address 127.2.70.62:41095)
0620 15:01:19.616063 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:19.616069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:19.617445 (+ 1376us) server_negotiation.cc:407] Connection header received
0620 15:01:19.619171 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:19.619175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:19.619239 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:19.619328 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:19.620402 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:19.621094 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:19.622847 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:19.623382 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:19.623482 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:19.625036 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:19.625044 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:19.625051 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:19.625075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:19.625116 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:19.625135 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:19.625235 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:19.625639 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 15:01:19.625728 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:01:19.627018 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:19.616398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:41095 (local address 127.2.70.2:58613)
0620 15:01:19.617295 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:19.617337 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:19.618910 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:19.619565 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:19.619574 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:19.619667 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:19.620256 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:19.620273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:19.621221 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:19.621225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:19.622699 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:19.622710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:19.623602 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:19.623609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:19.623885 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:19.624721 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:19.624724 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:19.624761 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:19.624837 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:19.625325 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:19.625334 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:19.625340 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:01:19.625635 (+ 295us) client_negotiation.cc:240] Negotiation successful
0620 15:01:19.626003 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":275,"threads_started":1}
I20260620 15:01:19.628381 11953 heartbeater.cc:347] Connected to a master server at 127.2.70.62:41095
I20260620 15:01:19.628685 11953 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:19.629398 11953 heartbeater.cc:511] Master 127.2.70.62:41095 requested a full tablet report, sending...
I20260620 15:01:19.630824 11636 ts_manager.cc:194] Registered new tserver with Master: c9683d7ff2f343c8a2777eaf2c94813a (127.2.70.2:45659)
I20260620 15:01:19.632004 11636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.2:58613
W20260620 15:01:19.970006 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:19.970316 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:19.970412 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:19.977509 11956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:19.977608 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:19.977710 11956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:01:19.986967 11956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:41095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:19.988194 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:19.989926 11956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:20.003492 11961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:20.003623 11964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:20.003602 11962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:20.003841 11956 server_base.cc:1034] running on GCE node
I20260620 15:01:20.004781 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:20.005844 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:20.007208 11956 hybrid_clock.cc:648] HybridClock initialized: now 1781967680007089 us; error 101 us; skew 500 ppm
I20260620 15:01:20.007587 11956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:01:20.009991 11956 webserver.cc:460] Webserver started at http://127.2.70.3:41915/ using document root <none> and password file <none>
I20260620 15:01:20.010730 11956 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:20.010902 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:20.011241 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:20.014642 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "25b10ae36a844abd982171aaaf2d0a8c"
format_stamp: "Formatted at 2026-06-20 15:01:20 on dist-test-slave-n5vv"
server_key: "d1ae86b1652b1ab84242e24a71bfedfa"
server_key_iv: "e1436501b041a3ed4abc9b52ec8b1686"
server_key_version: "encryptionkey@0"
I20260620 15:01:20.015425 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "25b10ae36a844abd982171aaaf2d0a8c"
format_stamp: "Formatted at 2026-06-20 15:01:20 on dist-test-slave-n5vv"
server_key: "d1ae86b1652b1ab84242e24a71bfedfa"
server_key_iv: "e1436501b041a3ed4abc9b52ec8b1686"
server_key_version: "encryptionkey@0"
I20260620 15:01:20.020613 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260620 15:01:20.024109 11970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:20.025417 11956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:01:20.025621 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "25b10ae36a844abd982171aaaf2d0a8c"
format_stamp: "Formatted at 2026-06-20 15:01:20 on dist-test-slave-n5vv"
server_key: "d1ae86b1652b1ab84242e24a71bfedfa"
server_key_iv: "e1436501b041a3ed4abc9b52ec8b1686"
server_key_version: "encryptionkey@0"
I20260620 15:01:20.025852 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:20.044029 11956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:20.044922 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:20.045207 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:20.046209 11956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:20.047807 11956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:20.047962 11956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:20.048086 11956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:20.048180 11956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:20.079037 11956 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:44861
I20260620 15:01:20.079070 12082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:44861 every 8 connection(s)
I20260620 15:01:20.080327 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:20.089452 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 11956
I20260620 15:01:20.089661 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:20.090188 2328 external_mini_cluster.cc:1428] Setting key fb84ac9b4f0130926868c8605b95c7d0
I20260620 15:01:20.093408 11824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:20.082985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:60897 (local address 127.2.70.62:41095)
0620 15:01:20.083197 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:20.083202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:20.084630 (+ 1428us) server_negotiation.cc:407] Connection header received
0620 15:01:20.086493 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:20.086499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:20.086590 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:20.086700 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:20.087730 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.088448 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.090511 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.090903 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.091025 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:20.092569 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:20.092574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:20.092577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:20.092593 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:20.092625 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:20.092636 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:20.092705 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:20.092998 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:01:20.093106 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:01:20.094558 12085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:20.083545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:41095 (local address 127.2.70.3:60897)
0620 15:01:20.084474 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:20.084522 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:20.086237 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:20.086926 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:20.086934 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:20.087021 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:20.087570 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:20.087587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.088603 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.088610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:20.090362 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:20.090384 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.091111 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.091118 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:20.091399 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:20.092241 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:20.092247 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:20.092304 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:20.092400 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:20.092733 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:20.092743 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:20.092751 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:01:20.092969 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 15:01:20.093378 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":296,"threads_started":1}
I20260620 15:01:20.095821 12083 heartbeater.cc:347] Connected to a master server at 127.2.70.62:41095
I20260620 15:01:20.096107 12083 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:20.096817 12083 heartbeater.cc:511] Master 127.2.70.62:41095 requested a full tablet report, sending...
I20260620 15:01:20.098104 11636 ts_manager.cc:194] Registered new tserver with Master: 25b10ae36a844abd982171aaaf2d0a8c (127.2.70.3:44861)
I20260620 15:01:20.098866 11636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.3:60897
I20260620 15:01:20.105294 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:01:20.111274 11824 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:20.109698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60932 (local address 127.2.70.62:41095)
0620 15:01:20.109885 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:20.109889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:20.110390 (+ 501us) server_negotiation.cc:407] Connection header received
0620 15:01:20.110547 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:20.110550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:20.110621 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:20.110722 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:20.111029 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60932: BlockingRecv error: recv got EOF from 127.0.0.1:60932 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:01:20.111726 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11695
I20260620 15:01:20.119916 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:20.112733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.2.70.1:42791)
0620 15:01:20.113149 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:20.113157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:20.113181 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:01:20.113233 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:20.113238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:20.113447 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:20.113626 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:20.113902 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.114842 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.116051 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.116588 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.116679 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:20.117694 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:20.117707 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:20.117725 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:20.118592 (+ 867us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:20.118650 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:20.118673 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:20.118790 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:20.119360 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 15:01:20.119523 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":191,"threads_started":1}
I20260620 15:01:20.121536 11817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:20.203105 11822 heartbeater.cc:503] Master 127.2.70.62:41095 was elected leader, sending a full tablet report...
I20260620 15:01:20.204109 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 11695
W20260620 15:01:20.218168 11621 connection.cc:535] server connection from 127.2.70.1:41043 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:01:20.218493 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11826
I20260620 15:01:20.226341 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:20.219343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.2.70.2:45659)
0620 15:01:20.219833 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:20.219840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:20.219866 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:01:20.219925 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:20.219931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:20.220154 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:20.220326 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:20.220740 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.221890 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.223100 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.223667 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.223801 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:20.224842 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:20.224851 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:20.224864 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:20.225454 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:20.225567 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:20.225589 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:20.225726 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:20.225880 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 15:01:20.226005 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":254,"threads_started":1}
I20260620 15:01:20.227586 11948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:20.308513 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 11826
I20260620 15:01:20.325089 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11956
I20260620 15:01:20.333266 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:20.326093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.2.70.3:44861)
0620 15:01:20.326731 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:20.326738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:20.326767 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:01:20.326826 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:20.326833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:20.327051 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:20.327220 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:20.327687 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.328721 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.330135 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:20.330694 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:20.330833 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:20.331935 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:20.331943 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:20.331956 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:20.332556 (+ 600us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:20.332599 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:20.332618 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:20.332708 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:20.332836 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 15:01:20.332952 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":333,"threads_started":1}
I20260620 15:01:20.334689 12078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:20.417970 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 11956
I20260620 15:01:20.433754 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 11605
I20260620 15:01:20.435045 11666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:20.539458 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 11605
2026-06-20T15:01:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2468 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T15:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:20Z Disabled control of system clock
I20260620 15:01:20.610419 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43929
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:43929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 15:01:20.928862 12102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:20.929150 12102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:20.929323 12102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:20.936450 12102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:20.936576 12102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:20.936676 12102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:20.936779 12102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:20.936856 12102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:20.946926 12102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:43929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:20.948261 12102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:20.950199 12102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:20.961985 12107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:20.962077 12110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:20.962050 12108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:20.962919 12102 server_base.cc:1034] running on GCE node
I20260620 15:01:20.963634 12102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:20.964921 12102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:20.966149 12102 hybrid_clock.cc:648] HybridClock initialized: now 1781967680966109 us; error 26 us; skew 500 ppm
I20260620 15:01:20.966560 12102 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:01:20.968897 12102 webserver.cc:460] Webserver started at http://127.2.70.62:38471/ using document root <none> and password file <none>
I20260620 15:01:20.969728 12102 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:20.969838 12102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:20.970161 12102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:20.972982 12102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "d9c6f384f34a434ebd7443c40b61f32d"
format_stamp: "Formatted at 2026-06-20 15:01:20 on dist-test-slave-n5vv"
server_key: "066385cf2380d27ae8b62604160824ad"
server_key_iv: "fd28b7dfe9882e55b2a55703e16f4ef0"
server_key_version: "encryptionkey@0"
I20260620 15:01:20.973836 12102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "d9c6f384f34a434ebd7443c40b61f32d"
format_stamp: "Formatted at 2026-06-20 15:01:20 on dist-test-slave-n5vv"
server_key: "066385cf2380d27ae8b62604160824ad"
server_key_iv: "fd28b7dfe9882e55b2a55703e16f4ef0"
server_key_version: "encryptionkey@0"
I20260620 15:01:20.978791 12102 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:01:20.982208 12116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:20.983544 12102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:20.983721 12102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "d9c6f384f34a434ebd7443c40b61f32d"
format_stamp: "Formatted at 2026-06-20 15:01:20 on dist-test-slave-n5vv"
server_key: "066385cf2380d27ae8b62604160824ad"
server_key_iv: "fd28b7dfe9882e55b2a55703e16f4ef0"
server_key_version: "encryptionkey@0"
I20260620 15:01:20.983899 12102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:20.989666 12102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:20.990381 12102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:20.990618 12102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:21.009533 12102 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:43929
I20260620 15:01:21.009574 12167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:43929 every 8 connection(s)
I20260620 15:01:21.010885 12102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:21.014798 12168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:21.020437 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 12102
I20260620 15:01:21.020639 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:01:21.021283 2328 external_mini_cluster.cc:1428] Setting key 2c49afe509aaf850c29c0c2e3c220e87
I20260620 15:01:21.026446 12168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d: Bootstrap starting.
I20260620 15:01:21.030577 12168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:21.031198 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:21.023449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.2.70.62:43929)
0620 15:01:21.024533 (+ 1084us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:21.024549 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:21.024596 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:01:21.026877 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:21.026912 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:21.027517 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:21.027956 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:21.029065 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:21.029073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:21.029088 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:21.029104 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:21.029172 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:21.029192 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:21.029407 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:21.029572 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 15:01:21.029844 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":301,"threads_started":1}
I20260620 15:01:21.031864 12168 log.cc:824] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:21.034955 12168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d: No bootstrap required, opened a new log
I20260620 15:01:21.042795 12168 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c6f384f34a434ebd7443c40b61f32d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43929 } }
I20260620 15:01:21.043206 12168 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:21.043322 12168 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c6f384f34a434ebd7443c40b61f32d, State: Initialized, Role: FOLLOWER
I20260620 15:01:21.043922 12168 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c6f384f34a434ebd7443c40b61f32d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43929 } }
I20260620 15:01:21.044210 12168 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:21.044363 12168 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:21.044549 12168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:21.047492 12168 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c6f384f34a434ebd7443c40b61f32d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43929 } }
I20260620 15:01:21.048022 12168 leader_election.cc:302] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9c6f384f34a434ebd7443c40b61f32d; no voters:
I20260620 15:01:21.048630 12168 leader_election.cc:288] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:21.048763 12173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:21.049618 12173 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [term 1 LEADER]: Becoming Leader. State: Replica: d9c6f384f34a434ebd7443c40b61f32d, State: Running, Role: LEADER
I20260620 15:01:21.050247 12173 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c6f384f34a434ebd7443c40b61f32d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43929 } }
I20260620 15:01:21.050947 12168 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:21.052824 12174 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9c6f384f34a434ebd7443c40b61f32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c6f384f34a434ebd7443c40b61f32d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43929 } } }
I20260620 15:01:21.053184 12174 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:21.054278 12175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9c6f384f34a434ebd7443c40b61f32d. Latest consensus state: current_term: 1 leader_uuid: "d9c6f384f34a434ebd7443c40b61f32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c6f384f34a434ebd7443c40b61f32d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43929 } } }
I20260620 15:01:21.054809 12175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:21.056139 12179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:21.061648 12179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:21.070271 12179 catalog_manager.cc:1347] Generated new cluster ID: a0675071cc634eb4a56389152bf9ffe2
I20260620 15:01:21.070420 12179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:21.080232 12179 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:21.080711 12179 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:21.091986 12179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9c6f384f34a434ebd7443c40b61f32d: Generated new TSK 0
I20260620 15:01:21.092864 12179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:01:21.106966 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43929
--builtin_ntp_servers=127.2.70.20:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 15:01:21.444221 12192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:21.444496 12192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:21.444651 12192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:21.452138 12192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:21.452246 12192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:21.452415 12192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:01:21.461980 12192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:21.463315 12192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:21.465091 12192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:21.477936 12197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:21.478082 12198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:21.478499 12200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:21.479066 12192 server_base.cc:1034] running on GCE node
I20260620 15:01:21.479748 12192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:21.480716 12192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:21.481940 12192 hybrid_clock.cc:648] HybridClock initialized: now 1781967681481903 us; error 53 us; skew 500 ppm
I20260620 15:01:21.482259 12192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:01:21.484386 12192 webserver.cc:460] Webserver started at http://127.2.70.1:44955/ using document root <none> and password file <none>
I20260620 15:01:21.485095 12192 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:21.485229 12192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:21.485641 12192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:21.490118 12192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "462e766898104bfa91d6ec90982089ed"
format_stamp: "Formatted at 2026-06-20 15:01:21 on dist-test-slave-n5vv"
server_key: "c7910be75381b92e5b2d5615f17e5c80"
server_key_iv: "f2509f018c882f72369b88e4b2060410"
server_key_version: "encryptionkey@0"
I20260620 15:01:21.490903 12192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "462e766898104bfa91d6ec90982089ed"
format_stamp: "Formatted at 2026-06-20 15:01:21 on dist-test-slave-n5vv"
server_key: "c7910be75381b92e5b2d5615f17e5c80"
server_key_iv: "f2509f018c882f72369b88e4b2060410"
server_key_version: "encryptionkey@0"
I20260620 15:01:21.496089 12192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:01:21.499616 12206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:21.500922 12192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:01:21.501103 12192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "462e766898104bfa91d6ec90982089ed"
format_stamp: "Formatted at 2026-06-20 15:01:21 on dist-test-slave-n5vv"
server_key: "c7910be75381b92e5b2d5615f17e5c80"
server_key_iv: "f2509f018c882f72369b88e4b2060410"
server_key_version: "encryptionkey@0"
I20260620 15:01:21.501330 12192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:21.507124 12192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:21.507905 12192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:21.508173 12192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:21.509057 12192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:21.510484 12192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:21.510618 12192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:21.510749 12192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:21.510852 12192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:21.544140 12192 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:46881
I20260620 15:01:21.544194 12318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:46881 every 8 connection(s)
I20260620 15:01:21.545545 12192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:21.553325 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 12192
I20260620 15:01:21.553543 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:21.553946 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:21.548174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:32943 (local address 127.2.70.62:43929)
0620 15:01:21.548551 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:21.548555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:21.549878 (+ 1323us) server_negotiation.cc:407] Connection header received
0620 15:01:21.551640 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:21.551644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:21.551693 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:21.551789 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:21.553114 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:21.553119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:21.553124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:21.553142 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:21.553187 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:21.553199 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:21.553328 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:21.553617 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:01:21.553683 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":184,"threads_started":1}
I20260620 15:01:21.554311 2328 external_mini_cluster.cc:1428] Setting key edbb21cd79ab930471077c3fdb5476aa
I20260620 15:01:21.554992 12322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:21.548787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43929 (local address 127.2.70.1:32943)
0620 15:01:21.549727 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:21.549768 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:21.551399 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:21.552049 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:21.552058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:21.552172 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:21.552804 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:21.552806 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:21.552843 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:21.552926 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:21.553303 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:21.553315 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:21.553322 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 15:01:21.553696 (+ 374us) client_negotiation.cc:240] Negotiation successful
0620 15:01:21.553947 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":244,"threads_started":1}
I20260620 15:01:21.557312 12319 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43929
I20260620 15:01:21.557372 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43929
--builtin_ntp_servers=127.2.70.20:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 15:01:21.557670 12319 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:21.558351 12319 heartbeater.cc:511] Master 127.2.70.62:43929 requested a full tablet report, sending...
I20260620 15:01:21.560415 12133 ts_manager.cc:194] Registered new tserver with Master: 462e766898104bfa91d6ec90982089ed (127.2.70.1:46881)
W20260620 15:01:21.883476 12323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:21.883747 12323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:21.883834 12323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:21.891321 12323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:21.891423 12323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:21.891533 12323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:01:21.900492 12323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:21.901768 12323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:21.903391 12323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:21.915853 12329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:21.915865 12328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:21.915940 12331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:21.916733 12323 server_base.cc:1034] running on GCE node
I20260620 15:01:21.917303 12323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:21.918324 12323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:21.919554 12323 hybrid_clock.cc:648] HybridClock initialized: now 1781967681919510 us; error 36 us; skew 500 ppm
I20260620 15:01:21.919874 12323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:01:21.922048 12323 webserver.cc:460] Webserver started at http://127.2.70.2:46091/ using document root <none> and password file <none>
I20260620 15:01:21.922701 12323 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:21.922833 12323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:21.923161 12323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:21.925912 12323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "308caba4e23f43e999c06cb684713e5d"
format_stamp: "Formatted at 2026-06-20 15:01:21 on dist-test-slave-n5vv"
server_key: "a6c37f2ace1316223aaa27a4e3a9a020"
server_key_iv: "c3439d5454d6f4ebd786cddb396e93a4"
server_key_version: "encryptionkey@0"
I20260620 15:01:21.926659 12323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "308caba4e23f43e999c06cb684713e5d"
format_stamp: "Formatted at 2026-06-20 15:01:21 on dist-test-slave-n5vv"
server_key: "a6c37f2ace1316223aaa27a4e3a9a020"
server_key_iv: "c3439d5454d6f4ebd786cddb396e93a4"
server_key_version: "encryptionkey@0"
I20260620 15:01:21.931514 12323 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 15:01:21.934798 12337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:21.935983 12323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:21.936160 12323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "308caba4e23f43e999c06cb684713e5d"
format_stamp: "Formatted at 2026-06-20 15:01:21 on dist-test-slave-n5vv"
server_key: "a6c37f2ace1316223aaa27a4e3a9a020"
server_key_iv: "c3439d5454d6f4ebd786cddb396e93a4"
server_key_version: "encryptionkey@0"
I20260620 15:01:21.936371 12323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:21.941864 12323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:21.942627 12323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:21.942903 12323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:21.943751 12323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:21.945104 12323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:21.945227 12323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:21.945369 12323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:21.945461 12323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:21.977630 12323 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:34035
I20260620 15:01:21.977648 12449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:34035 every 8 connection(s)
I20260620 15:01:21.979001 12323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:21.979604 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 12323
I20260620 15:01:21.979769 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:21.980348 2328 external_mini_cluster.cc:1428] Setting key 8ce95500e4393c0810800d8ec9838a0a
I20260620 15:01:21.983475 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43929
--builtin_ntp_servers=127.2.70.20:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 15:01:21.987920 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:21.981853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:51879 (local address 127.2.70.62:43929)
0620 15:01:21.982055 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:21.982061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:21.983733 (+ 1672us) server_negotiation.cc:407] Connection header received
0620 15:01:21.985466 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:21.985474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:21.985584 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:21.985687 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:21.987074 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:21.987082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:21.987089 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:21.987113 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:21.987153 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:21.987170 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:21.987302 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:21.987503 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 15:01:21.987572 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:01:21.988951 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:21.982601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43929 (local address 127.2.70.2:51879)
0620 15:01:21.983547 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:21.983623 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:21.985177 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:21.985900 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:21.985908 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:21.986004 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:21.986759 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:21.986764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:21.986806 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:21.986890 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:21.987265 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:21.987272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:21.987300 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:01:21.987610 (+ 310us) client_negotiation.cc:240] Negotiation successful
0620 15:01:21.987812 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":298,"threads_started":1}
I20260620 15:01:21.990401 12450 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43929
I20260620 15:01:21.990722 12450 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:21.991303 12450 heartbeater.cc:511] Master 127.2.70.62:43929 requested a full tablet report, sending...
I20260620 15:01:21.992739 12133 ts_manager.cc:194] Registered new tserver with Master: 308caba4e23f43e999c06cb684713e5d (127.2.70.2:34035)
W20260620 15:01:22.313033 12453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:22.313302 12453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:22.313544 12453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:22.320818 12453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:22.320912 12453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:22.321090 12453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:01:22.330209 12453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:22.331496 12453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:22.333257 12453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:22.346061 12458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:22.346163 12461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:22.346139 12459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:22.347225 12453 server_base.cc:1034] running on GCE node
I20260620 15:01:22.347824 12453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:22.348754 12453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:22.349987 12453 hybrid_clock.cc:648] HybridClock initialized: now 1781967682349929 us; error 45 us; skew 500 ppm
I20260620 15:01:22.350386 12453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:01:22.352718 12453 webserver.cc:460] Webserver started at http://127.2.70.3:39581/ using document root <none> and password file <none>
I20260620 15:01:22.353391 12453 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:22.353561 12453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:22.353889 12453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:22.356665 12453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "85bed029e9e04e2eb601cedb614aac6a"
format_stamp: "Formatted at 2026-06-20 15:01:22 on dist-test-slave-n5vv"
server_key: "6f91d7351755e1a0b5d51902d210658e"
server_key_iv: "0d276932f875ecde50ccad662a3e0a12"
server_key_version: "encryptionkey@0"
I20260620 15:01:22.357465 12453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "85bed029e9e04e2eb601cedb614aac6a"
format_stamp: "Formatted at 2026-06-20 15:01:22 on dist-test-slave-n5vv"
server_key: "6f91d7351755e1a0b5d51902d210658e"
server_key_iv: "0d276932f875ecde50ccad662a3e0a12"
server_key_version: "encryptionkey@0"
I20260620 15:01:22.362602 12453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 15:01:22.366225 12467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:22.367617 12453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:22.367797 12453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "85bed029e9e04e2eb601cedb614aac6a"
format_stamp: "Formatted at 2026-06-20 15:01:22 on dist-test-slave-n5vv"
server_key: "6f91d7351755e1a0b5d51902d210658e"
server_key_iv: "0d276932f875ecde50ccad662a3e0a12"
server_key_version: "encryptionkey@0"
I20260620 15:01:22.368026 12453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:22.373733 12453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:22.374517 12453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:22.374804 12453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:22.375806 12453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:22.377153 12453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:22.377275 12453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:22.377413 12453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:22.377527 12453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:22.411252 12453 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:39271
I20260620 15:01:22.411309 12579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:39271 every 8 connection(s)
I20260620 15:01:22.412572 12453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:22.416527 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 12453
I20260620 15:01:22.416692 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:22.417188 2328 external_mini_cluster.cc:1428] Setting key 45bbfd1f3d7fcb8a9fff3328f83a4fa4
I20260620 15:01:22.420966 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:22.415212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:38979 (local address 127.2.70.62:43929)
0620 15:01:22.415387 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:22.415392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:22.416884 (+ 1492us) server_negotiation.cc:407] Connection header received
0620 15:01:22.418671 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:22.418676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:22.418743 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:22.418867 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:22.420241 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:22.420251 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:22.420255 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:22.420269 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:22.420298 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:22.420309 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:22.420428 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:22.420631 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 15:01:22.420698 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:01:22.422004 12582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:22.415758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43929 (local address 127.2.70.3:38979)
0620 15:01:22.416694 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:22.416751 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:22.418381 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:22.419096 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:22.419104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:22.419200 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:22.419946 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:01:22.419949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:22.419987 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:01:22.420078 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:22.420405 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:22.420413 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:22.420421 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 15:01:22.420710 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 15:01:22.420970 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":262,"threads_started":1}
I20260620 15:01:22.423203 12580 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43929
I20260620 15:01:22.423491 12580 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:22.424069 12580 heartbeater.cc:511] Master 127.2.70.62:43929 requested a full tablet report, sending...
I20260620 15:01:22.425377 12133 ts_manager.cc:194] Registered new tserver with Master: 85bed029e9e04e2eb601cedb614aac6a (127.2.70.3:39271)
I20260620 15:01:22.432371 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 15:01:22.438552 12321 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:01:22.436840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39688 (local address 127.2.70.62:43929)
0620 15:01:22.437012 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:22.437019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:22.437659 (+ 640us) server_negotiation.cc:407] Connection header received
0620 15:01:22.437790 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:22.437794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:22.437843 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:22.437923 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:22.438221 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39688: BlockingRecv error: recv got EOF from 127.0.0.1:39688 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:01:22.439352 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 12192
I20260620 15:01:22.442898 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:22.440310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51850 (local address 127.2.70.1:46881)
0620 15:01:22.440862 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:22.440868 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:22.440892 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:01:22.440947 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:22.440953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:22.441161 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:22.441282 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:22.442229 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:22.442234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:22.442246 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:22.442262 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:22.442308 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:22.442321 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:22.442423 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:22.442498 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 15:01:22.442616 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":276,"threads_started":1}
I20260620 15:01:22.444060 12314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:22.528745 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 12192
I20260620 15:01:22.544835 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 12323
I20260620 15:01:22.548427 12591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:22.545769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42456 (local address 127.2.70.2:34035)
0620 15:01:22.546242 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:22.546249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:22.546276 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:01:22.546333 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:22.546340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:22.546561 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:22.546732 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:22.547712 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:22.547718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:22.547730 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:22.547745 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:22.547782 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:22.547795 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:22.547942 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:22.548073 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 15:01:22.548173 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260620 15:01:22.549654 12445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:22.630837 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 12323
I20260620 15:01:22.646366 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 12453
I20260620 15:01:22.650172 12593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:22.647336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.2.70.3:39271)
0620 15:01:22.647958 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:22.647965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:22.647996 (+ 31us) server_negotiation.cc:407] Connection header received
0620 15:01:22.648075 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:22.648082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:22.648293 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:22.648428 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:22.649459 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:22.649465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:22.649476 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:01:22.649576 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:22.649621 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:01:22.649638 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:22.649767 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:22.649825 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 15:01:22.649900 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":326,"threads_started":1}
I20260620 15:01:22.651619 12575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:22.734012 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 12453
I20260620 15:01:22.749670 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 12102
I20260620 15:01:22.750959 12163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:22.850929 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 12102
2026-06-20T15:01:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2311 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:22 dist-test-slave-n5vv krb5kdc[12598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:22 dist-test-slave-n5vv krb5kdc[12598](info): set up 2 sockets
Jun 20 15:01:22 dist-test-slave-n5vv krb5kdc[12598](info): commencing operation
krb5kdc: starting...
W20260620 15:01:23.950346 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:01:24 dist-test-slave-n5vv krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:01:24.186715 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46575
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:46575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:24.510532 12611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:24.510975 12611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:24.511143 12611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:24.519331 12611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:01:24.519479 12611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:24.519585 12611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:24.519695 12611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:01:24.519763 12611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:01:24.529894 12611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:46575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:24.531210 12611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:24.532531 12611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:24.544236 12616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:24.544206 12617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:24.544166 12619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:24.544968 12611 server_base.cc:1034] running on GCE node
I20260620 15:01:24.545855 12611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:24.547227 12611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:24.548501 12611 hybrid_clock.cc:648] HybridClock initialized: now 1781967684548529 us; error 126 us; skew 500 ppm
I20260620 15:01:24.548880 12611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:24 dist-test-slave-n5vv krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967684, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:24.551954 12611 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:01:24.553411 12611 webserver.cc:460] Webserver started at http://127.2.70.62:36241/ using document root <none> and password file <none>
I20260620 15:01:24.554090 12611 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:24.554236 12611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:24.554546 12611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:24.557332 12611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "add7d08117de43ce9883fbf7ad485478"
format_stamp: "Formatted at 2026-06-20 15:01:24 on dist-test-slave-n5vv"
server_key: "ec99930dbe7d7d4727f06b4b40c1083f"
server_key_iv: "cf0205a0a05bc560b53cf70f25c7ce84"
server_key_version: "encryptionkey@0"
I20260620 15:01:24.558141 12611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "add7d08117de43ce9883fbf7ad485478"
format_stamp: "Formatted at 2026-06-20 15:01:24 on dist-test-slave-n5vv"
server_key: "ec99930dbe7d7d4727f06b4b40c1083f"
server_key_iv: "cf0205a0a05bc560b53cf70f25c7ce84"
server_key_version: "encryptionkey@0"
I20260620 15:01:24.563010 12611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 15:01:24.566269 12626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:24.567528 12611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:24.567700 12611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "add7d08117de43ce9883fbf7ad485478"
format_stamp: "Formatted at 2026-06-20 15:01:24 on dist-test-slave-n5vv"
server_key: "ec99930dbe7d7d4727f06b4b40c1083f"
server_key_iv: "cf0205a0a05bc560b53cf70f25c7ce84"
server_key_version: "encryptionkey@0"
I20260620 15:01:24.567889 12611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:24.584858 12611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:24.589178 12611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:24.589470 12611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:24.607995 12611 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:46575
I20260620 15:01:24.608031 12677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:46575 every 8 connection(s)
I20260620 15:01:24.609349 12611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:01:24.612864 12678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:24.618321 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 12611
I20260620 15:01:24.618533 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:01:24.619177 2328 external_mini_cluster.cc:1428] Setting key c6b3b9279457576d0dda41616aeb2215
I20260620 15:01:24.624181 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478: Bootstrap starting.
I20260620 15:01:24.627969 12678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:24.629231 12678 log.cc:824] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:24.632465 12678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478: No bootstrap required, opened a new log
Jun 20 15:01:24 dist-test-slave-n5vv krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:01:24.642602 12678 raft_consensus.cc:348] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add7d08117de43ce9883fbf7ad485478" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46575 } }
I20260620 15:01:24.643064 12678 raft_consensus.cc:374] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:24.643168 12678 raft_consensus.cc:729] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add7d08117de43ce9883fbf7ad485478, State: Initialized, Role: FOLLOWER
I20260620 15:01:24.643821 12678 consensus_queue.cc:260] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add7d08117de43ce9883fbf7ad485478" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46575 } }
I20260620 15:01:24.644096 12678 raft_consensus.cc:388] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:01:24.644236 12678 raft_consensus.cc:482] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:01:24.644400 12681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:24.621221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.2.70.62:46575)
0620 15:01:24.622098 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:24.622114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:24.622166 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:01:24.624507 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:24.624536 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:24.625050 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:24.625498 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:24.626922 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:24.628939 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:24.630729 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:24.631744 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:24.631933 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:24.635948 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:24.635957 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:24.635976 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:24.636032 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:24.639942 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:24.640607 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:24.640616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:24.640626 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:24.640864 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:24.641215 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:24.641220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:24.641223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:24.641941 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:24.642275 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:24.642544 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:01:24.642992 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":393,"threads_started":1}
I20260620 15:01:24.644424 12678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:24.647156 12678 raft_consensus.cc:504] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add7d08117de43ce9883fbf7ad485478" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46575 } }
I20260620 15:01:24.647662 12678 leader_election.cc:302] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: add7d08117de43ce9883fbf7ad485478; no voters:
I20260620 15:01:24.648164 12678 leader_election.cc:288] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:01:24.648306 12683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:24.648962 12683 raft_consensus.cc:686] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [term 1 LEADER]: Becoming Leader. State: Replica: add7d08117de43ce9883fbf7ad485478, State: Running, Role: LEADER
I20260620 15:01:24.649590 12683 consensus_queue.cc:237] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add7d08117de43ce9883fbf7ad485478" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46575 } }
I20260620 15:01:24.650256 12678 sys_catalog.cc:564] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:01:24.652264 12684 sys_catalog.cc:455] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "add7d08117de43ce9883fbf7ad485478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add7d08117de43ce9883fbf7ad485478" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46575 } } }
I20260620 15:01:24.652590 12684 sys_catalog.cc:458] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [sys.catalog]: This master's current role is: LEADER
I20260620 15:01:24.652066 12685 sys_catalog.cc:455] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [sys.catalog]: SysCatalogTable state changed. Reason: New leader add7d08117de43ce9883fbf7ad485478. Latest consensus state: current_term: 1 leader_uuid: "add7d08117de43ce9883fbf7ad485478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "add7d08117de43ce9883fbf7ad485478" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46575 } } }
I20260620 15:01:24.653031 12685 sys_catalog.cc:458] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478 [sys.catalog]: This master's current role is: LEADER
W20260620 15:01:24.661928 12699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:01:24.662053 12699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:01:24.662210 12688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:01:24.664394 12688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:01:24.673597 12688 catalog_manager.cc:1347] Generated new cluster ID: fda4407cadd14e21b6a62db88770224d
I20260620 15:01:24.673714 12688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:01:24.687073 12688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:01:24.688488 12688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:01:24.694384 12688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478: Generated new TSK 0
I20260620 15:01:24.695456 12688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:01:24.752615 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46575
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:25.079945 12704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:25.080219 12704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:25.080379 12704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:25.087555 12704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:25.087661 12704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:25.087873 12704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:01:25.097872 12704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:25.099112 12704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:25.100421 12704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:25.113721 12712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.113752 12710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.114223 12709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:25.114478 12704 server_base.cc:1034] running on GCE node
I20260620 15:01:25.115104 12704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:25.116011 12704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:25.117229 12704 hybrid_clock.cc:648] HybridClock initialized: now 1781967685117196 us; error 33 us; skew 500 ppm
I20260620 15:01:25.117663 12704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:25 dist-test-slave-n5vv krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:25.120832 12704 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:01:25.122329 12704 webserver.cc:460] Webserver started at http://127.2.70.1:46517/ using document root <none> and password file <none>
I20260620 15:01:25.122982 12704 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:25.123085 12704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:25.123318 12704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:25.126503 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "a0a4417e144d4fe0b413a8efc4922436"
format_stamp: "Formatted at 2026-06-20 15:01:25 on dist-test-slave-n5vv"
server_key: "a6a21455fa43f56a256c2f661548cff9"
server_key_iv: "b88e91040331a8e8b08eb021248d1fc2"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.127300 12704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "a0a4417e144d4fe0b413a8efc4922436"
format_stamp: "Formatted at 2026-06-20 15:01:25 on dist-test-slave-n5vv"
server_key: "a6a21455fa43f56a256c2f661548cff9"
server_key_iv: "b88e91040331a8e8b08eb021248d1fc2"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.132747 12704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:01:25.136302 12719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.137852 12704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:25.137979 12704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "a0a4417e144d4fe0b413a8efc4922436"
format_stamp: "Formatted at 2026-06-20 15:01:25 on dist-test-slave-n5vv"
server_key: "a6a21455fa43f56a256c2f661548cff9"
server_key_iv: "b88e91040331a8e8b08eb021248d1fc2"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.138151 12704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:25.160826 12704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:25.164614 12704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:25.164850 12704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:25.165740 12704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:25.167016 12704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:25.167086 12704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.167270 12704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:25.167333 12704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.201474 12704 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:40503
I20260620 15:01:25.201576 12831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:40503 every 8 connection(s)
I20260620 15:01:25.202857 12704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:01:25.207041 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 12704
I20260620 15:01:25.207232 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:01:25.207944 2328 external_mini_cluster.cc:1428] Setting key 8c883e7fd069df400f46054c3f62e5d3
Jun 20 15:01:25 dist-test-slave-n5vv krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:01:25.223480 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.205567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:34061 (local address 127.2.70.62:46575)
0620 15:01:25.206045 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:25.206049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:25.207117 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 15:01:25.209184 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:25.209190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:25.209279 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:25.209413 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:25.211039 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:25.211779 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:25.213340 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:25.213841 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:25.213979 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:25.217644 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:25.217649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:25.217652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:25.217683 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:25.220234 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:25.221161 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:25.221165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:25.221167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:25.221274 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:25.221818 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:25.221821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:25.221823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:25.222080 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:25.222186 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:25.223059 (+ 873us) server_negotiation.cc:299] Negotiation successful
0620 15:01:25.223219 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":234,"threads_started":1}
I20260620 15:01:25.225030 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.206164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46575 (local address 127.2.70.1:34061)
0620 15:01:25.206956 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.206995 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:25.208907 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:25.209723 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:25.209735 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:25.210284 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:25.210861 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:25.210877 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:25.211948 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:25.211959 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:25.213191 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:25.213202 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:25.214067 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:25.214075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:25.214319 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:25.215025 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:25.215047 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:25.217351 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:25.220378 (+ 3027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:25.220390 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:25.220410 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:25.221041 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:25.221394 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:25.221402 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:25.221408 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:25.221705 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:25.222220 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:25.222231 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:25.222672 (+ 441us) client_negotiation.cc:769] Sending connection context
0620 15:01:25.222963 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 15:01:25.223564 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":376,"threads_started":1}
I20260620 15:01:25.226682 12832 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46575
I20260620 15:01:25.226986 12832 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:25.227712 12832 heartbeater.cc:511] Master 127.2.70.62:46575 requested a full tablet report, sending...
I20260620 15:01:25.229809 12643 ts_manager.cc:194] Registered new tserver with Master: a0a4417e144d4fe0b413a8efc4922436 (127.2.70.1:40503)
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:01:25.231374 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:34061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:01:25.260008 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46575
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:25.583062 12838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:25.583345 12838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:25.583530 12838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:25.591035 12838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:25.591145 12838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:25.591355 12838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:01:25.601296 12838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:25.602624 12838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:25.603943 12838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:25.617105 12843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.617132 12844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:25.617807 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:25.625156 12838 server_base.cc:1034] running on GCE node
I20260620 15:01:25.625857 12838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:25.626809 12838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:25.628046 12838 hybrid_clock.cc:648] HybridClock initialized: now 1781967685628008 us; error 52 us; skew 500 ppm
I20260620 15:01:25.628365 12838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:25 dist-test-slave-n5vv krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:25.631628 12838 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:01:25.633097 12838 webserver.cc:460] Webserver started at http://127.2.70.2:46595/ using document root <none> and password file <none>
I20260620 15:01:25.633885 12838 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:25.634034 12838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:25.634342 12838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:25.637233 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "269f81c3092a49069b90235a925eb0ec"
format_stamp: "Formatted at 2026-06-20 15:01:25 on dist-test-slave-n5vv"
server_key: "15d00ee21781b6e74b54d50405f5b1d2"
server_key_iv: "b85ea1bf05be8201bd90f5e874a33994"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.638082 12838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "269f81c3092a49069b90235a925eb0ec"
format_stamp: "Formatted at 2026-06-20 15:01:25 on dist-test-slave-n5vv"
server_key: "15d00ee21781b6e74b54d50405f5b1d2"
server_key_iv: "b85ea1bf05be8201bd90f5e874a33994"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.643172 12838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:01:25.646612 12853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.647925 12838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:01:25.648121 12838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "269f81c3092a49069b90235a925eb0ec"
format_stamp: "Formatted at 2026-06-20 15:01:25 on dist-test-slave-n5vv"
server_key: "15d00ee21781b6e74b54d50405f5b1d2"
server_key_iv: "b85ea1bf05be8201bd90f5e874a33994"
server_key_version: "encryptionkey@0"
I20260620 15:01:25.648324 12838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:25.667500 12838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:25.672228 12838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:25.672538 12838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:25.673471 12838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:25.674870 12838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:25.674978 12838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.675137 12838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:25.675218 12838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:25.707336 12838 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:37657
I20260620 15:01:25.707402 12965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:37657 every 8 connection(s)
I20260620 15:01:25.708654 12838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:01:25.711982 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 12838
I20260620 15:01:25.712164 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:01:25.712613 2328 external_mini_cluster.cc:1428] Setting key 3ffa24c83dab9ccd617eff2e2fdf9bf8
Jun 20 15:01:25 dist-test-slave-n5vv krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967685, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:01:25.730443 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.711194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:60627 (local address 127.2.70.62:46575)
0620 15:01:25.711365 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:25.711369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:25.712674 (+ 1305us) server_negotiation.cc:407] Connection header received
0620 15:01:25.715026 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:25.715032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:25.715107 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:25.715222 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:25.716641 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:25.717762 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:25.719308 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:25.720051 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:25.720195 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:25.723779 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:25.723788 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:25.723796 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:25.723841 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:25.727329 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:25.728004 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:25.728010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:25.728015 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:25.728153 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:25.728540 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:25.728548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:25.728554 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:25.728928 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:25.729081 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:25.729757 (+ 676us) server_negotiation.cc:299] Negotiation successful
0620 15:01:25.730007 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:01:25.731102 12968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:25.711682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46575 (local address 127.2.70.2:60627)
0620 15:01:25.712477 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:25.712548 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:25.714763 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:25.715414 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:25.715422 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:25.715917 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:25.716461 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:25.716478 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:25.717901 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:25.717905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:25.719162 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:25.719173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:25.720252 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:25.720257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:25.720450 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:25.721167 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:25.721192 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:25.723551 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:25.727464 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:25.727471 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:25.727484 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:25.727880 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:25.728259 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:25.728262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:25.728264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:25.728427 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:25.729051 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:25.729057 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:25.729351 (+ 294us) client_negotiation.cc:769] Sending connection context
0620 15:01:25.729640 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 15:01:25.730026 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":341,"threads_started":1}
I20260620 15:01:25.732622 12966 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46575
I20260620 15:01:25.732975 12966 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:25.733780 12966 heartbeater.cc:511] Master 127.2.70.62:46575 requested a full tablet report, sending...
I20260620 15:01:25.735217 12643 ts_manager.cc:194] Registered new tserver with Master: 269f81c3092a49069b90235a925eb0ec (127.2.70.2:37657)
I20260620 15:01:25.736006 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:60627
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:01:25.764485 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46575
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:35581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:01:26.084538 12971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:01:26.084852 12971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:01:26.085035 12971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:01:26.092255 12971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:01:26.092409 12971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:01:26.092602 12971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:01:26.102340 12971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:35581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:01:26.103696 12971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:01:26.105067 12971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:01:26.118556 12979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:26.118631 12976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:01:26.118628 12977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:01:26.119766 12971 server_base.cc:1034] running on GCE node
I20260620 15:01:26.120409 12971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:01:26.121440 12971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:01:26.122766 12971 hybrid_clock.cc:648] HybridClock initialized: now 1781967686122713 us; error 49 us; skew 500 ppm
I20260620 15:01:26.123142 12971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:01:26 dist-test-slave-n5vv krb5kdc[12598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:01:26.126377 12971 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:01:26.127713 12971 webserver.cc:460] Webserver started at http://127.2.70.3:33707/ using document root <none> and password file <none>
I20260620 15:01:26.128338 12971 fs_manager.cc:359] Metadata directory not provided
I20260620 15:01:26.128436 12971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:01:26.128677 12971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:01:26.131645 12971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "419a9bd70d4e4af89938e30a49e6b3b6"
format_stamp: "Formatted at 2026-06-20 15:01:26 on dist-test-slave-n5vv"
server_key: "d0406ed24e677079b1601c32025f77d1"
server_key_iv: "66ed3c6e566b3c75843d9d333910d261"
server_key_version: "encryptionkey@0"
I20260620 15:01:26.132342 12971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "419a9bd70d4e4af89938e30a49e6b3b6"
format_stamp: "Formatted at 2026-06-20 15:01:26 on dist-test-slave-n5vv"
server_key: "d0406ed24e677079b1601c32025f77d1"
server_key_iv: "66ed3c6e566b3c75843d9d333910d261"
server_key_version: "encryptionkey@0"
I20260620 15:01:26.137246 12971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:01:26.140440 12986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:26.141881 12971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:01:26.142035 12971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "419a9bd70d4e4af89938e30a49e6b3b6"
format_stamp: "Formatted at 2026-06-20 15:01:26 on dist-test-slave-n5vv"
server_key: "d0406ed24e677079b1601c32025f77d1"
server_key_iv: "66ed3c6e566b3c75843d9d333910d261"
server_key_version: "encryptionkey@0"
I20260620 15:01:26.142277 12971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:01:26.160256 12971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:01:26.164319 12971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:01:26.164567 12971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:01:26.165627 12971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:01:26.166980 12971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:01:26.167055 12971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:26.167194 12971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:01:26.167268 12971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:01:26.199957 12971 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:42627
I20260620 15:01:26.199990 13098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:42627 every 8 connection(s)
I20260620 15:01:26.201370 12971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:01:26.205777 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 12971
I20260620 15:01:26.205965 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:01:26.206396 2328 external_mini_cluster.cc:1428] Setting key fa6a44f8644d5a539b4a361828755dfb
Jun 20 15:01:26 dist-test-slave-n5vv krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967686, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:01:26.221936 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.204236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:54289 (local address 127.2.70.62:46575)
0620 15:01:26.204462 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.204468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.205825 (+ 1357us) server_negotiation.cc:407] Connection header received
0620 15:01:26.208054 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.208059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.208160 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.208271 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:26.209973 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.210828 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.212398 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.212791 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.212927 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.216503 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:26.216509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:26.216514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:26.216545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:26.218941 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.219666 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.219676 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.219679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.219796 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.220222 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.220227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.220230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.220486 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:26.220641 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.221362 (+ 721us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.221602 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 15:01:26.222856 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.204776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46575 (local address 127.2.70.3:54289)
0620 15:01:26.205665 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.205718 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:26.207772 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:26.208520 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:26.208528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:26.209141 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:01:26.209805 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.209825 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.210995 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.211000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.212249 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.212260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.213012 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.213022 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.213262 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.213937 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:01:26.213958 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:01:26.216296 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:01:26.219092 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:26.219102 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:26.219117 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:26.219533 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:26.219908 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:01:26.219915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:01:26.219918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:01:26.220100 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:01:26.220613 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:01:26.220621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:01:26.220950 (+ 329us) client_negotiation.cc:769] Sending connection context
0620 15:01:26.221216 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 15:01:26.221785 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":316,"threads_started":1}
I20260620 15:01:26.224154 13099 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46575
I20260620 15:01:26.224414 13099 heartbeater.cc:464] Registering TS with master...
I20260620 15:01:26.225098 13099 heartbeater.cc:511] Master 127.2.70.62:46575 requested a full tablet report, sending...
I20260620 15:01:26.226521 12643 ts_manager.cc:194] Registered new tserver with Master: 419a9bd70d4e4af89938e30a49e6b3b6 (127.2.70.3:42627)
I20260620 15:01:26.227334 12643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:54289
I20260620 15:01:26.234057 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:01:26.234387 12832 heartbeater.cc:503] Master 127.2.70.62:46575 was elected leader, sending a full tablet report...
I20260620 15:01:26.250875 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.238711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.2.70.62:46575)
0620 15:01:26.238862 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.238866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.239307 (+ 441us) server_negotiation.cc:407] Connection header received
0620 15:01:26.239438 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.239441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.239494 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.239586 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:26.240903 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.241708 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.242983 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.243371 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.243553 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.245789 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:26.245793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:26.245797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:26.245826 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:26.248223 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.248843 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.248850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.248858 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.248973 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.249342 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.249348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.249352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.249703 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:26.249951 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.250333 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.250564 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:01:26.255049 12643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:01:26.261924 12643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:01:26.300123 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.286819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.2.70.1:40503)
0620 15:01:26.292658 (+ 5839us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.292665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.292689 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:01:26.292771 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.292776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.292973 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.293199 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:26.293849 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.295468 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.297440 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.298900 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.299066 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.299305 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.299591 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.299749 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5698,"thread_start_us":5561,"threads_started":1}
I20260620 15:01:26.300134 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.286599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40503 (local address 127.0.0.1:47788)
0620 15:01:26.292109 (+ 5510us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.292131 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:26.292214 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:26.293193 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:26.293200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:26.293234 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:26.293636 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.293652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.295600 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.295605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.297250 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.297263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.299108 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.299118 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.299312 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.299379 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 15:01:26.299539 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 15:01:26.299752 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5398,"thread_start_us":386,"threads_started":1}
I20260620 15:01:26.302526 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.273260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:37657 (local address 127.0.0.1:36166)
0620 15:01:26.287551 (+ 14291us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.287614 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:26.287849 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:26.294923 (+ 7074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:26.294928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:26.294957 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:26.295315 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.295324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.297278 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.297287 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.299749 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.299760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.301658 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.301665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.301855 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.301879 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:01:26.301940 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 15:01:26.302089 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13936,"mutex_wait_us":61,"thread_start_us":18160,"threads_started":1}
I20260620 15:01:26.302584 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.274579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35546 (local address 127.2.70.3:42627)
0620 15:01:26.293978 (+ 19399us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.293989 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.294022 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:01:26.294117 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.294127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.294368 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.294569 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:26.295221 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.296977 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.299901 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.301335 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.301529 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.301778 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.302074 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.302237 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19217,"thread_start_us":18926,"threads_started":1}
I20260620 15:01:26.302574 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.274503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:42627 (local address 127.0.0.1:35546)
0620 15:01:26.291199 (+ 16696us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.291222 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:26.291338 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:26.294570 (+ 3232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:26.294575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:26.294601 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:26.295010 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.295025 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.297155 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.297162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.299708 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.299727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.301683 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.301692 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.301883 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.301915 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:01:26.302010 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:01:26.302139 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16558,"mutex_wait_us":27,"thread_start_us":17155,"threads_started":1}
I20260620 15:01:26.303375 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.273469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36166 (local address 127.2.70.2:37657)
0620 15:01:26.294331 (+ 20862us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.294337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.294366 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:01:26.294457 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.294461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.294699 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.294911 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:26.295527 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.297127 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.299904 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.301335 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.302775 (+ 1440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.302992 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.303044 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.303153 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20674,"thread_start_us":20569,"threads_started":1}
I20260620 15:01:26.304694 12767 tablet_service.cc:1467] Processing CreateTablet for tablet e5e6d51ec10449fd806048fe57e2e159 (DEFAULT_TABLE table=test-table [id=eee60f3bb7da4cc3ba83bc88bdee7176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:26.306110 12767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e6d51ec10449fd806048fe57e2e159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:26.306282 13034 tablet_service.cc:1467] Processing CreateTablet for tablet e5e6d51ec10449fd806048fe57e2e159 (DEFAULT_TABLE table=test-table [id=eee60f3bb7da4cc3ba83bc88bdee7176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:26.307377 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e6d51ec10449fd806048fe57e2e159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:26.308789 12901 tablet_service.cc:1467] Processing CreateTablet for tablet e5e6d51ec10449fd806048fe57e2e159 (DEFAULT_TABLE table=test-table [id=eee60f3bb7da4cc3ba83bc88bdee7176]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:01:26.310101 12901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5e6d51ec10449fd806048fe57e2e159. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:01:26.318362 13117 tablet_bootstrap.cc:492] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Bootstrap starting.
I20260620 15:01:26.322038 13117 tablet_bootstrap.cc:654] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:26.322176 13118 tablet_bootstrap.cc:492] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: Bootstrap starting.
I20260620 15:01:26.323380 13117 log.cc:824] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:26.325918 13118 tablet_bootstrap.cc:654] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:26.326092 13120 tablet_bootstrap.cc:492] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Bootstrap starting.
I20260620 15:01:26.327363 13118 log.cc:824] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:26.329195 13120 tablet_bootstrap.cc:654] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Neither blocks nor log segments found. Creating new log.
I20260620 15:01:26.329982 13117 tablet_bootstrap.cc:492] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: No bootstrap required, opened a new log
I20260620 15:01:26.330262 13117 ts_tablet_manager.cc:1397] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260620 15:01:26.330508 13120 log.cc:824] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Log is configured to *not* fsync() on all Append() calls
I20260620 15:01:26.332139 13118 tablet_bootstrap.cc:492] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: No bootstrap required, opened a new log
I20260620 15:01:26.332397 13118 ts_tablet_manager.cc:1397] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 15:01:26.332966 13120 tablet_bootstrap.cc:492] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: No bootstrap required, opened a new log
I20260620 15:01:26.333209 13120 ts_tablet_manager.cc:1397] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260620 15:01:26.340833 13120 raft_consensus.cc:348] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.340885 13117 raft_consensus.cc:348] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.341244 13120 raft_consensus.cc:374] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:26.341405 13117 raft_consensus.cc:374] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:26.341418 13120 raft_consensus.cc:729] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269f81c3092a49069b90235a925eb0ec, State: Initialized, Role: FOLLOWER
I20260620 15:01:26.341552 13117 raft_consensus.cc:729] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 419a9bd70d4e4af89938e30a49e6b3b6, State: Initialized, Role: FOLLOWER
I20260620 15:01:26.342324 13117 consensus_queue.cc:260] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.342445 13120 consensus_queue.cc:260] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.343214 13118 raft_consensus.cc:348] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.343690 12966 heartbeater.cc:503] Master 127.2.70.62:46575 was elected leader, sending a full tablet report...
I20260620 15:01:26.343904 13118 raft_consensus.cc:374] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:01:26.344084 13118 raft_consensus.cc:729] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0a4417e144d4fe0b413a8efc4922436, State: Initialized, Role: FOLLOWER
I20260620 15:01:26.344393 13120 ts_tablet_manager.cc:1428] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260620 15:01:26.344843 13118 consensus_queue.cc:260] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.348366 13118 ts_tablet_manager.cc:1428] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20260620 15:01:26.349440 13099 heartbeater.cc:503] Master 127.2.70.62:46575 was elected leader, sending a full tablet report...
I20260620 15:01:26.350512 13117 ts_tablet_manager.cc:1428] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Time spent starting tablet: real 0.020s user 0.008s sys 0.012s
W20260620 15:01:26.453126 13100 tablet.cc:2369] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:01:26.455197 12833 tablet.cc:2369] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:01:26.460598 12967 tablet.cc:2369] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:01:26.465941 13125 raft_consensus.cc:482] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:01:26.466187 13125 raft_consensus.cc:504] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.467763 13125 leader_election.cc:288] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269f81c3092a49069b90235a925eb0ec (127.2.70.2:37657), 419a9bd70d4e4af89938e30a49e6b3b6 (127.2.70.3:42627)
I20260620 15:01:26.474928 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.468154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:33037 (local address 127.2.70.3:42627)
0620 15:01:26.468330 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.468339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.468817 (+ 478us) server_negotiation.cc:407] Connection header received
0620 15:01:26.469031 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.469039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.469131 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.469228 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:26.469940 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.470901 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.472977 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.473964 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.474062 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.474135 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.474556 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.474686 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 15:01:26.474988 13126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.468010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:42627 (local address 127.2.70.1:33037)
0620 15:01:26.468702 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.468720 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:26.468877 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:26.469285 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:26.469291 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:26.469330 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:26.469772 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.469784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.471018 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.471024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.472825 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.472836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.474169 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.474176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.474348 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.474383 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:01:26.474491 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:01:26.474631 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":3,"thread_start_us":398,"threads_started":1}
I20260620 15:01:26.475589 13127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.468124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:37657 (local address 127.2.70.1:49543)
0620 15:01:26.469448 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.469461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 15:01:26.469637 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:01:26.469972 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:01:26.469981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:01:26.470028 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:01:26.470389 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.470399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.471649 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.471653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.473284 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:01:26.473299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.474626 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.474632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:01:26.474797 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.474830 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:01:26.474930 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:01:26.475270 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1199,"thread_start_us":1001,"threads_started":1}
I20260620 15:01:26.475600 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.468136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:49543 (local address 127.2.70.2:37657)
0620 15:01:26.468355 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.468362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.469576 (+ 1214us) server_negotiation.cc:407] Connection header received
0620 15:01:26.469769 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.469774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.469854 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.469945 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:01:26.470534 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.471524 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.473431 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.474404 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.474523 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.474611 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.474974 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.475288 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:01:26.476066 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e6d51ec10449fd806048fe57e2e159" candidate_uuid: "a0a4417e144d4fe0b413a8efc4922436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" is_pre_election: true
I20260620 15:01:26.476553 13054 raft_consensus.cc:2454] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0a4417e144d4fe0b413a8efc4922436 in term 0.
I20260620 15:01:26.476589 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e6d51ec10449fd806048fe57e2e159" candidate_uuid: "a0a4417e144d4fe0b413a8efc4922436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269f81c3092a49069b90235a925eb0ec" is_pre_election: true
I20260620 15:01:26.477129 12921 raft_consensus.cc:2454] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0a4417e144d4fe0b413a8efc4922436 in term 0.
I20260620 15:01:26.477224 12721 leader_election.cc:302] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419a9bd70d4e4af89938e30a49e6b3b6, a0a4417e144d4fe0b413a8efc4922436; no voters:
I20260620 15:01:26.477665 13125 raft_consensus.cc:2781] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:01:26.477841 13125 raft_consensus.cc:482] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:01:26.477977 13125 raft_consensus.cc:3037] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:26.480330 13125 raft_consensus.cc:504] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.480959 13125 leader_election.cc:288] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [CANDIDATE]: Term 1 election: Requested vote from peers 269f81c3092a49069b90235a925eb0ec (127.2.70.2:37657), 419a9bd70d4e4af89938e30a49e6b3b6 (127.2.70.3:42627)
I20260620 15:01:26.481390 12921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e6d51ec10449fd806048fe57e2e159" candidate_uuid: "a0a4417e144d4fe0b413a8efc4922436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269f81c3092a49069b90235a925eb0ec"
I20260620 15:01:26.481439 13054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5e6d51ec10449fd806048fe57e2e159" candidate_uuid: "a0a4417e144d4fe0b413a8efc4922436" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "419a9bd70d4e4af89938e30a49e6b3b6"
I20260620 15:01:26.481639 12921 raft_consensus.cc:3037] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:26.481665 13054 raft_consensus.cc:3037] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:01:26.483734 13054 raft_consensus.cc:2454] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a4417e144d4fe0b413a8efc4922436 in term 1.
I20260620 15:01:26.483739 12921 raft_consensus.cc:2454] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0a4417e144d4fe0b413a8efc4922436 in term 1.
I20260620 15:01:26.484133 12721 leader_election.cc:302] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 419a9bd70d4e4af89938e30a49e6b3b6, a0a4417e144d4fe0b413a8efc4922436; no voters:
I20260620 15:01:26.484424 13125 raft_consensus.cc:2781] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:01:26.484877 13125 raft_consensus.cc:686] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 1 LEADER]: Becoming Leader. State: Replica: a0a4417e144d4fe0b413a8efc4922436, State: Running, Role: LEADER
I20260620 15:01:26.485337 13125 consensus_queue.cc:237] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } }
I20260620 15:01:26.490252 12643 catalog_manager.cc:5495] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0a4417e144d4fe0b413a8efc4922436 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "a0a4417e144d4fe0b413a8efc4922436" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0a4417e144d4fe0b413a8efc4922436" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40503 } health_report { overall_health: HEALTHY } } }
I20260620 15:01:26.523682 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.518341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47802 (local address 127.2.70.1:40503)
0620 15:01:26.518565 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.518573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.518644 (+ 71us) server_negotiation.cc:407] Connection header received
0620 15:01:26.518876 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.518884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.518965 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.519122 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:01:26.519481 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.520192 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.521538 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.521953 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.522086 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.522603 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:01:26.523102 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:01:26.523271 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.523328 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.523410 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:01:26.531843 12921 raft_consensus.cc:1261] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 1 FOLLOWER]: Refusing update from remote peer a0a4417e144d4fe0b413a8efc4922436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:26.532783 13054 raft_consensus.cc:1261] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 1 FOLLOWER]: Refusing update from remote peer a0a4417e144d4fe0b413a8efc4922436: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:01:26.532826 13128 consensus_queue.cc:1035] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269f81c3092a49069b90235a925eb0ec" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 37657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:26.533622 13125 consensus_queue.cc:1035] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [LEADER]: Connected to new peer: Peer: permanent_uuid: "419a9bd70d4e4af89938e30a49e6b3b6" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 42627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:01:26.545626 13132 mvcc.cc:205] Tried to move back new op lower bound from 7298939644024324096 to 7298939643848093696. Current Snapshot: MvccSnapshot[applied={T|T < 7298939644024324096}]
I20260620 15:01:26.551564 13133 mvcc.cc:205] Tried to move back new op lower bound from 7298939644024324096 to 7298939643848093696. Current Snapshot: MvccSnapshot[applied={T|T < 7298939644024324096}]
I20260620 15:01:26.565310 12643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:26.565605 12643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:01:26.568815 12643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P add7d08117de43ce9883fbf7ad485478: Sending DeleteTablet for 3 replicas of tablet e5e6d51ec10449fd806048fe57e2e159
I20260620 15:01:26.569968 12901 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e6d51ec10449fd806048fe57e2e159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:26 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:36166
I20260620 15:01:26.570010 13034 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e6d51ec10449fd806048fe57e2e159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:26 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:35546
I20260620 15:01:26.570303 12767 tablet_service.cc:1514] Processing DeleteTablet for tablet e5e6d51ec10449fd806048fe57e2e159 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:01:26 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:47788
I20260620 15:01:26.572140 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 12704
I20260620 15:01:26.583961 13143 tablet_replica.cc:330] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: stopping tablet replica
Jun 20 15:01:26 dist-test-slave-n5vv krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:01:26.584589 13143 raft_consensus.cc:2229] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:01:26.586167 13142 tablet_replica.cc:330] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: stopping tablet replica
I20260620 15:01:26.585426 13143 raft_consensus.cc:2258] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:26.587455 13141 tablet_replica.cc:330] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: stopping tablet replica
I20260620 15:01:26.587864 13141 raft_consensus.cc:2229] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:26.588385 13141 raft_consensus.cc:2258] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:26.589921 13143 ts_tablet_manager.cc:1905] T e5e6d51ec10449fd806048fe57e2e159 P a0a4417e144d4fe0b413a8efc4922436: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:26.590557 13141 ts_tablet_manager.cc:1905] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:26.593535 13142 raft_consensus.cc:2229] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:01:26.594123 13142 raft_consensus.cc:2258] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:01:26.594197 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.573248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.2.70.1:40503)
0620 15:01:26.573464 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.573473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.573644 (+ 171us) server_negotiation.cc:407] Connection header received
0620 15:01:26.573953 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.573962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.574088 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.574178 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:26.576280 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.577577 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.579424 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.580179 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.580275 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.585867 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:26.585882 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:26.585900 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:26.585951 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:26.590273 (+ 4322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.591343 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.591352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.591356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.591536 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.592049 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.592054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.592058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.592505 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:26.592811 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.593421 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.593807 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:01:26.595907 12827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:26.596347 13142 ts_tablet_manager.cc:1905] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:01:26.598038 13141 ts_tablet_manager.cc:1918] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:26.598217 13141 log.cc:1196] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/e5e6d51ec10449fd806048fe57e2e159
I20260620 15:01:26.598727 13141 ts_tablet_manager.cc:1939] T e5e6d51ec10449fd806048fe57e2e159 P 269f81c3092a49069b90235a925eb0ec: Deleting consensus metadata
I20260620 15:01:26.600332 12627 catalog_manager.cc:4841] TS 269f81c3092a49069b90235a925eb0ec (127.2.70.2:37657): tablet e5e6d51ec10449fd806048fe57e2e159 (table test-table [id=eee60f3bb7da4cc3ba83bc88bdee7176]) successfully deleted
I20260620 15:01:26.605867 13142 ts_tablet_manager.cc:1918] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:01:26.606045 13142 log.cc:1196] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/e5e6d51ec10449fd806048fe57e2e159
I20260620 15:01:26.606556 13142 ts_tablet_manager.cc:1939] T e5e6d51ec10449fd806048fe57e2e159 P 419a9bd70d4e4af89938e30a49e6b3b6: Deleting consensus metadata
I20260620 15:01:26.608862 12628 catalog_manager.cc:4841] TS 419a9bd70d4e4af89938e30a49e6b3b6 (127.2.70.3:42627): tablet e5e6d51ec10449fd806048fe57e2e159 (table test-table [id=eee60f3bb7da4cc3ba83bc88bdee7176]) successfully deleted
I20260620 15:01:26.700824 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 12704
W20260620 15:01:26.722368 12630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.1:40503 (error 108)
I20260620 15:01:26.723179 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 12838
I20260620 15:01:26.724738 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.724164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40503 (local address 127.0.0.1:47830)
0620 15:01:26.724425 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.724556 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260620 15:01:26.725028 12630 catalog_manager.cc:4584] TS a0a4417e144d4fe0b413a8efc4922436 (127.2.70.1:40503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e5e6d51ec10449fd806048fe57e2e159: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40503: connect: Connection refused (error 111)
Jun 20 15:01:26 dist-test-slave-n5vv krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:01:26.740372 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.724062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.2.70.2:37657)
0620 15:01:26.724259 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.724266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.724400 (+ 134us) server_negotiation.cc:407] Connection header received
0620 15:01:26.724730 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.724736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.724837 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.724925 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:26.726434 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.727615 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.728988 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.729725 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.729840 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.734175 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:26.734189 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:26.734208 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:26.734247 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:26.737952 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.738683 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.738689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.738692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.738808 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.739116 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.739119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.739121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.739483 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:26.739694 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.739960 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.740119 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:01:26.741096 12961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:26.778815 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.778251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40503 (local address 127.0.0.1:47838)
0620 15:01:26.778500 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.778613 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 15:01:26.779875 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.779475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40503 (local address 127.0.0.1:47850)
0620 15:01:26.779631 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.779734 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 15:01:26.837606 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 12838
I20260620 15:01:26.840266 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.839866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40503 (local address 127.0.0.1:47864)
0620 15:01:26.840050 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.840133 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:01:26.841218 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.840884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40503 (local address 127.0.0.1:47880)
0620 15:01:26.841023 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.841098 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:01:26.855163 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 12971
Jun 20 15:01:26 dist-test-slave-n5vv krb5kdc[12598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:01:26.870678 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.855952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35556 (local address 127.2.70.3:42627)
0620 15:01:26.856153 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:01:26.856160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:01:26.856182 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:01:26.856387 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:01:26.856398 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:01:26.856461 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:01:26.856572 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:01:26.857899 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.858609 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.859904 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:01:26.860293 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:01:26.860433 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:01:26.864806 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:01:26.864821 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:01:26.864839 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:01:26.864890 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:01:26.867753 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.868500 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.868509 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.868512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.868660 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:01:26.869057 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:01:26.869064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:01:26.869071 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:01:26.869416 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:01:26.869828 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0620 15:01:26.870132 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 15:01:26.870347 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:01:26.871377 13094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:26.921905 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.921384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40503 (local address 127.0.0.1:47896)
0620 15:01:26.921659 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.921742 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260620 15:01:26.922974 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:01:26.922635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40503 (local address 127.0.0.1:47906)
0620 15:01:26.922809 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 15:01:26.922875 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 15:01:26.959580 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 12971
I20260620 15:01:26.975979 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 12611
I20260620 15:01:26.977116 12673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:01:27.083541 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 12611
2026-06-20T15:01:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4234 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:01:27 dist-test-slave-n5vv krb5kdc[13152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:01:27 dist-test-slave-n5vv krb5kdc[13152](info): set up 2 sockets
Jun 20 15:01:27 dist-test-slave-n5vv krb5kdc[13152](info): commencing operation
krb5kdc: starting...
W20260620 15:01:28.190613 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.007s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:01:28 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:01:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.70.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.70.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/rangeradmin_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/rangeradmin_127.2.70.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.70.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.70.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/rangerlookup_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/rangerlookup_127.2.70.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.70.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.70.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/HTTP_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/HTTP_127.2.70.20@KRBTEST.COM.keytab.
I20260620 15:01:28.512822 2328 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskK9gJB7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/postgres -l logfile start
2026-06-20 15:01:30.968 UTC [13182] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 15:01:30.968 UTC [13182] LOG: listening on IPv4 address "127.2.70.20", port 37645
2026-06-20 15:01:30.971 UTC [13182] LOG: listening on Unix socket "/tmp/.s.PGSQL.37645"
2026-06-20 15:01:30.993 UTC [13185] LOG: database system was shut down at 2026-06-20 15:01:29 UTC
2026-06-20 15:01:30.997 UTC [13182] LOG: database system is ready to accept connections
I20260620 15:01:31.974022 2328 mini_postgres.cc:96] Postgres bound to 37645
2026-06-20 15:01:31.991 UTC [13193] FATAL: database "slave" does not exist
127.2.70.20:37645 - accepting connections
I20260620 15:01:31.992276 2328 mini_ranger.cc:161] Starting Ranger...
I20260620 15:01:32.019199 2328 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 15:01:32.522982 2328 mini_ranger.cc:87] Created ranger Postgres database
I20260620 15:01:32.523156 2328 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-admin
2026-06-20 15:01:32,808 [I] DB FLAVOR :POSTGRES
2026-06-20 15:01:32,809 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:01:32,809 [I] Checking connection..
2026-06-20 15:01:32,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 15:01:33,050 [I] Checking connection passed.
2026-06-20 15:01:33,050 [I] --------- Verifying version history table ---------
2026-06-20 15:01:33,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:01:33,297 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:01:33,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:01:33,534 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:01:33,535 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 15:01:33,535 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 15:01:33.764 UTC [13257] WARNING: there is no transaction in progress
2026-06-20 15:01:33,771 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 15:01:33,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:01:34,026 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 15:01:34,026 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:01:34,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 15:01:34,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 15:01:34,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n5vv.c.gcp-upstream.internal','N') ;"
2026-06-20 15:01:34,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:01:35,042 [I] Table x_portal_user does not exist in database ranger
2026-06-20 15:01:35,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:01:35,286 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 15:01:35,286 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:01:35,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:01:35,763 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 15:01:35,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 15:01:36.720 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.736 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.752 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.767 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.786 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.806 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.858 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.865 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.879 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.889 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.899 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.912 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.922 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.928 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.934 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:36.940 UTC [13365] WARNING: there is no transaction in progress
2026-06-20 15:01:37,559 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 15:01:37,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 15:01:37,800 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 15:01:37,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:01:38,034 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 15:01:38,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:01:38,275 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 15:01:38,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:01:38,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:01:38,781 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n5vv.c.gcp-upstream.internal';"
2026-06-20 15:01:39,043 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 15:01:39,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 15:01:39,306 [I] DB_PATCHES have already been applied
I20260620 15:01:39.311864 2328 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-admin:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskK9gJB7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 15:01:39.312026 2328 mini_ranger.cc:193] Using host: 127.2.70.20
I20260620 15:01:39.326735 2328 mini_ranger.cc:227] Ranger admin URL: http://127.2.70.20:58007
Jun 20, 2026 3:01:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:01:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:01:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:01:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:01:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:01:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:01:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 15:01:40.462442 2328 mini_ranger.cc:160] Time spent starting Ranger: real 8.470s user 0.000s sys 0.041s
Jun 20, 2026 3:02:00 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:02:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 3:02:00 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:02:00 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:02:01 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:02:01.820567 2328 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.70.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.70.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/rangerkms_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/rangerkms_127.2.70.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.70.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.70.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.70.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/HTTP_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.70.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/HTTP_127.2.70.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:02:01 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967721, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:02:01.983649 2328 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:02:02.004292 2328 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:02:02.897861 2328 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:02:02.897974 2328 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms
I20260620 15:02:02.898149 2328 mini_ranger_kms.cc:227] Using postgres at 127.2.70.20:37645
2026-06-20 15:02:03,315 [I] DB FLAVOR :POSTGRES
2026-06-20 15:02:03,316 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:02:03,316 [I] Checking connection
2026-06-20 15:02:03,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:02:03,545 [I] connection success
2026-06-20 15:02:03,545 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:02:03,546 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:02:03,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:02:03,789 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:02:03,790 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:02:03,790 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:02:03,790 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:37645/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:02:04,065 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:02:04.519646 2328 mini_ranger_kms.cc:310] Created kms service
I20260620 15:02:04.691689 2328 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:02:04.735558 2328 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:02:04 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967721, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:02:05.129463 2328 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:02:05.129824 2328 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms:/tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:02:05.129894 2328 mini_ranger_kms.cc:240] Using host: 127.2.70.20
I20260620 15:02:05.147280 2328 mini_ranger_kms.cc:276] Ranger KMS PID: 13545
I20260620 15:02:05.147490 2328 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.70.20:52215
Jun 20, 2026 3:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:02:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 15:02:07.489666 2328 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.506s user 0.113s sys 0.324s
I20260620 15:02:07.489930 2328 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:02:07.490181 2328 mini_ranger_kms.cc:401] 127.2.70.20:52215/kms/v1/keys
Jun 20 15:02:07 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967727, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 15:02:07 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967727, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.20@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
Jun 20 15:02:08 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967727, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.20@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
Jun 20 15:02:10 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:02:10.628778 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46233
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:46233
--ranger_config_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:52215/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:02:10.974543 13596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:10.974818 13596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:10.974954 13596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:10.982197 13596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:02:10.982297 13596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:10.982362 13596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:10.982460 13596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:02:10.982537 13596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:02:10.993366 13596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.70.20:52215/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:46233
--ranger_config_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:02:10.995162 13596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:10.996563 13596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:11.010113 13601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:11.010164 13602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:11.010776 13604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:11.011168 13596 server_base.cc:1034] running on GCE node
I20260620 15:02:11.011988 13596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:11.013525 13596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:11.014747 13596 hybrid_clock.cc:648] HybridClock initialized: now 1781967731014695 us; error 44 us; skew 500 ppm
I20260620 15:02:11.015118 13596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:02:11 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:11.019021 13596 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:02:11.020725 13596 webserver.cc:460] Webserver started at http://127.2.70.62:40893/ using document root <none> and password file <none>
I20260620 15:02:11.021582 13596 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:11.021703 13596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:11.022049 13596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:02:11 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967731, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:02:11.084733 13596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "3a090ba2a3ce47509065e0ab0ba8ad71"
format_stamp: "Formatted at 2026-06-20 15:02:11 on dist-test-slave-n5vv"
server_key: "e4455f422eddf346d54512ed143ae438"
server_key_iv: "8f1314143d84461fb459b0469cb9eb69"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:11.085665 13596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "3a090ba2a3ce47509065e0ab0ba8ad71"
format_stamp: "Formatted at 2026-06-20 15:02:11 on dist-test-slave-n5vv"
server_key: "e4455f422eddf346d54512ed143ae438"
server_key_iv: "8f1314143d84461fb459b0469cb9eb69"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:11.091755 13596 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260620 15:02:11.111282 13611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:11.112877 13596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260620 15:02:11.113067 13596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "3a090ba2a3ce47509065e0ab0ba8ad71"
format_stamp: "Formatted at 2026-06-20 15:02:11 on dist-test-slave-n5vv"
server_key: "e4455f422eddf346d54512ed143ae438"
server_key_iv: "8f1314143d84461fb459b0469cb9eb69"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:11.113301 13596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:11.133361 13596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:11.137859 13596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:11.138154 13596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:11.158610 13596 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:46233
I20260620 15:02:11.158632 13662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:46233 every 8 connection(s)
I20260620 15:02:11.159981 13596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:02:11.164240 13663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:11.165233 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 13596
I20260620 15:02:11.165429 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:02:11.166213 2328 external_mini_cluster.cc:1428] Setting key ce6f756804f7d96cff6f38c73e10ce12
I20260620 15:02:11.181742 13663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71: Bootstrap starting.
I20260620 15:02:11.185267 13663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:11.187034 13663 log.cc:824] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:11.190303 13663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71: No bootstrap required, opened a new log
Jun 20 15:02:11 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:02:11.199152 13663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a090ba2a3ce47509065e0ab0ba8ad71" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46233 } }
I20260620 15:02:11.199569 13663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:11.199697 13663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a090ba2a3ce47509065e0ab0ba8ad71, State: Initialized, Role: FOLLOWER
I20260620 15:02:11.200799 13663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a090ba2a3ce47509065e0ab0ba8ad71" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46233 } }
I20260620 15:02:11.201054 13663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:02:11.201205 13663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:02:11.201404 13663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:11.205279 13663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a090ba2a3ce47509065e0ab0ba8ad71" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46233 } }
I20260620 15:02:11.206067 13663 leader_election.cc:302] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a090ba2a3ce47509065e0ab0ba8ad71; no voters:
I20260620 15:02:11.206842 13663 leader_election.cc:288] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:02:11.206969 13668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:11.207904 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:11.168331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48744 (local address 127.2.70.62:46233)
0620 15:02:11.183020 (+ 14689us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:11.183037 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:11.183087 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:02:11.185536 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:11.185567 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:11.186080 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:11.186512 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:11.188283 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:11.190222 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:11.192119 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:11.193200 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:11.193392 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:11.198141 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:11.198151 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:11.198174 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:11.198225 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:11.202545 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:11.203408 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:11.203417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:11.203427 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:11.203622 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:11.204083 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:11.204087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:11.204091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:11.204805 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:11.205094 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:11.205670 (+ 576us) server_negotiation.cc:299] Negotiation successful
0620 15:02:11.206219 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14442,"thread_start_us":17038,"threads_started":1}
I20260620 15:02:11.214774 13663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:02:11.216032 13668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [term 1 LEADER]: Becoming Leader. State: Replica: 3a090ba2a3ce47509065e0ab0ba8ad71, State: Running, Role: LEADER
I20260620 15:02:11.216656 13668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a090ba2a3ce47509065e0ab0ba8ad71" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46233 } }
I20260620 15:02:11.226785 13669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a090ba2a3ce47509065e0ab0ba8ad71. Latest consensus state: current_term: 1 leader_uuid: "3a090ba2a3ce47509065e0ab0ba8ad71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a090ba2a3ce47509065e0ab0ba8ad71" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46233 } } }
I20260620 15:02:11.227151 13669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:11.227293 13670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a090ba2a3ce47509065e0ab0ba8ad71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a090ba2a3ce47509065e0ab0ba8ad71" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46233 } } }
I20260620 15:02:11.227674 13670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71 [sys.catalog]: This master's current role is: LEADER
I20260620 15:02:11.238687 13678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:02:11.239399 13663 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:02:11.252497 13678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:02:11.276016 13678 catalog_manager.cc:1347] Generated new cluster ID: 38a52aad8f4045858667b4d093caeae3
I20260620 15:02:11.276134 13678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:02:11.287247 13678 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:02:11.289206 13678 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:02:11.297678 13678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71: Generated new TSK 0
I20260620 15:02:11.298799 13678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:02:12.113704 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:52215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46233
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:02:12 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:02:12.533360 13708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:12.533872 13708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:12.534457 13708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:12.544575 13708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:12.544749 13708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:12.544935 13708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:02:12.555845 13708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.70.20:52215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:02:12.557353 13708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:12.558835 13708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:12.577663 13718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:12.578419 13719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:12.583047 13721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:12.583480 13708 server_base.cc:1034] running on GCE node
I20260620 15:02:12.584295 13708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:12.585676 13708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:12.587003 13708 hybrid_clock.cc:648] HybridClock initialized: now 1781967732586887 us; error 103 us; skew 500 ppm
I20260620 15:02:12.587438 13708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:02:12 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:12.592414 13708 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:02:12.597874 13708 webserver.cc:460] Webserver started at http://127.2.70.1:39891/ using document root <none> and password file <none>
I20260620 15:02:12.599231 13708 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:12.599423 13708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:12.599916 13708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:02:12 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:02:12.621924 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "1eaa1295ac6546638fa1917bd17baa9c"
format_stamp: "Formatted at 2026-06-20 15:02:12 on dist-test-slave-n5vv"
server_key: "7e7650ac19c80a1671456cb4ca88e5f4"
server_key_iv: "3aa99df5d2ddbf10a98d4ec20afbf5f4"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:12.623219 13708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "1eaa1295ac6546638fa1917bd17baa9c"
format_stamp: "Formatted at 2026-06-20 15:02:12 on dist-test-slave-n5vv"
server_key: "7e7650ac19c80a1671456cb4ca88e5f4"
server_key_iv: "3aa99df5d2ddbf10a98d4ec20afbf5f4"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:12.631560 13708 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260620 15:02:12.651235 13728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:12.652974 13708 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:02:12.653174 13708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "1eaa1295ac6546638fa1917bd17baa9c"
format_stamp: "Formatted at 2026-06-20 15:02:12 on dist-test-slave-n5vv"
server_key: "7e7650ac19c80a1671456cb4ca88e5f4"
server_key_iv: "3aa99df5d2ddbf10a98d4ec20afbf5f4"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:12.653425 13708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:12.695876 13708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:12.701522 13708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:12.701864 13708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:12.703521 13708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:02:12.705394 13708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:02:12.705565 13708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:12.705765 13708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:02:12.706315 13708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:02:12 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:02:12.874958 13708 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:46121
I20260620 15:02:12.875026 13840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:46121 every 8 connection(s)
I20260620 15:02:12.876643 13708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:02:12.884609 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 13708
I20260620 15:02:12.884809 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:02:12.885634 2328 external_mini_cluster.cc:1428] Setting key 545c7a8633e2203c5b6f469ee0a2cfde
Jun 20 15:02:12 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:02:12.926928 13843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:12.890825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46233 (local address 127.2.70.1:46529)
0620 15:02:12.901193 (+ 10368us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:12.901270 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:12.904899 (+ 3629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:12.906969 (+ 2070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:12.906981 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:12.907590 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:12.908470 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:12.908496 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:12.910243 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:12.910249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:12.912443 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:12.912468 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:12.913756 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:12.913764 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:12.914047 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:12.914931 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:12.914961 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:12.918340 (+ 3379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:12.922418 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:12.922427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:12.922446 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:12.922987 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:12.923391 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:12.923396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:12.923399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:12.923627 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:12.924294 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:12.924302 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:12.924715 (+ 413us) client_negotiation.cc:769] Sending connection context
0620 15:02:12.925021 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 15:02:12.925571 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9985,"thread_start_us":10508,"threads_started":1}
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
I20260620 15:02:12.927948 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:12.891537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:46529 (local address 127.2.70.62:46233)
0620 15:02:12.906369 (+ 14832us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:12.906375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:12.906395 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:02:12.906458 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:12.906463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:12.906539 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:12.908657 (+ 2118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:12.908835 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:12.910038 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:12.912661 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:12.913313 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:12.913435 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:12.918648 (+ 5213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:12.918655 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:12.918660 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:12.918714 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:12.922251 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:12.923135 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:12.923140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:12.923144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:12.923275 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:12.923747 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:12.923751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:12.923754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:12.924092 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:12.924211 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:12.927419 (+ 3208us) server_negotiation.cc:299] Negotiation successful
0620 15:02:12.927636 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14698,"thread_start_us":17372,"threads_started":1}
I20260620 15:02:12.929833 13841 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46233
I20260620 15:02:12.930177 13841 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
I20260620 15:02:12.931053 13841 heartbeater.cc:511] Master 127.2.70.62:46233 requested a full tablet report, sending...
I20260620 15:02:12.933722 13628 ts_manager.cc:194] Registered new tserver with Master: 1eaa1295ac6546638fa1917bd17baa9c (127.2.70.1:46121)
I20260620 15:02:12.935832 13628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:46529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:02:12.965019 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:52215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46233
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:02:13 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967732, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
W20260620 15:02:13.433665 13847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:13.434145 13847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:13.434312 13847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:13.447618 13847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:13.447819 13847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:13.448019 13847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:02:13.465101 13847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.70.20:52215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:02:13.466913 13847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:13.468652 13847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:13.485306 13862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:13.485328 13861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:13.489593 13864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:13.491411 13847 server_base.cc:1034] running on GCE node
I20260620 15:02:13.492115 13847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:13.493304 13847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:13.494582 13847 hybrid_clock.cc:648] HybridClock initialized: now 1781967733494515 us; error 79 us; skew 500 ppm
I20260620 15:02:13.495024 13847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:02:13 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:13.499284 13847 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:02:13.502782 13847 webserver.cc:460] Webserver started at http://127.2.70.2:34935/ using document root <none> and password file <none>
I20260620 15:02:13.503695 13847 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:13.503852 13847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:13.504202 13847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:02:13 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:02:13.523675 13847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "354e12e61d32474daeff95115886c11d"
format_stamp: "Formatted at 2026-06-20 15:02:13 on dist-test-slave-n5vv"
server_key: "9ecd6f0c615d59c51fe2bb422605ae81"
server_key_iv: "7d2ed2bd54f93baf828807f0fc22def6"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:13.524854 13847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "354e12e61d32474daeff95115886c11d"
format_stamp: "Formatted at 2026-06-20 15:02:13 on dist-test-slave-n5vv"
server_key: "9ecd6f0c615d59c51fe2bb422605ae81"
server_key_iv: "7d2ed2bd54f93baf828807f0fc22def6"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:13.531888 13847 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260620 15:02:13.551472 13871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:13.553215 13847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260620 15:02:13.553440 13847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "354e12e61d32474daeff95115886c11d"
format_stamp: "Formatted at 2026-06-20 15:02:13 on dist-test-slave-n5vv"
server_key: "9ecd6f0c615d59c51fe2bb422605ae81"
server_key_iv: "7d2ed2bd54f93baf828807f0fc22def6"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:13.553764 13847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:13.576146 13847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:13.580878 13847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:13.581214 13847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:13.582410 13847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:02:13.584028 13847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:02:13.584146 13847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:13.584309 13847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:02:13.584385 13847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:13.646581 13847 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:39003
I20260620 15:02:13.646596 13983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:39003 every 8 connection(s)
I20260620 15:02:13.648036 13847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:02:13.650313 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 13847
I20260620 15:02:13.650554 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:02:13.651207 2328 external_mini_cluster.cc:1428] Setting key b4e745264b7773ef35c891680c2f84ab
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Jun 20 15:02:13 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967733, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:02:13.708403 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:13.666520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:44781 (local address 127.2.70.62:46233)
0620 15:02:13.689180 (+ 22660us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:13.689186 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:13.689208 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:02:13.689272 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:13.689277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:13.689414 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:13.689565 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:13.691845 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:13.692923 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:13.695330 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:13.695929 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:13.696024 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:13.701002 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:13.701009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:13.701014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:13.701062 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:13.704546 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:13.705540 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:13.705546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:13.705550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:13.705700 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:13.706255 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:13.706260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:13.706263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:13.706630 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:13.706745 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:13.707778 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0620 15:02:13.708010 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22527,"thread_start_us":22394,"threads_started":1}
I20260620 15:02:13.710898 13988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:13.667199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46233 (local address 127.2.70.2:44781)
0620 15:02:13.685523 (+ 18324us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:13.685583 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:13.687994 (+ 2411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:13.690015 (+ 2021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:13.690028 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:13.690746 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:13.691631 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:13.691655 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:13.693104 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:13.693110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:13.695166 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:13.695181 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:13.696342 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:13.696348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:13.696650 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:13.697463 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:13.697518 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:13.700742 (+ 3224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:13.704742 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:13.704751 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:13.704772 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:13.705356 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:13.705865 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:13.705870 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:13.705873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:13.706131 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:13.706851 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:13.706859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:13.707299 (+ 440us) client_negotiation.cc:769] Sending connection context
0620 15:02:13.708894 (+ 1595us) client_negotiation.cc:240] Negotiation successful
0620 15:02:13.709458 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12762,"thread_start_us":17021,"threads_started":1}
I20260620 15:02:13.712838 13984 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46233
I20260620 15:02:13.713191 13984 heartbeater.cc:464] Registering TS with master...
I20260620 15:02:13.714052 13984 heartbeater.cc:511] Master 127.2.70.62:46233 requested a full tablet report, sending...
I20260620 15:02:13.715694 13628 ts_manager.cc:194] Registered new tserver with Master: 354e12e61d32474daeff95115886c11d (127.2.70.2:39003)
I20260620 15:02:13.716763 13628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:44781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:02:13.730050 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:52215/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46233
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:02:13.939761 13841 heartbeater.cc:503] Master 127.2.70.62:46233 was elected leader, sending a full tablet report...
W20260620 15:02:14.181733 13990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:02:14.182001 13990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:02:14.182101 13990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:02:14.189111 13990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:02:14.189220 13990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:02:14.189332 13990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:02:14.199384 13990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.70.20:52215/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46233
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:02:14.200613 13990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:02:14.201949 13990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:02:14.215822 13998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:14.215883 13996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:02:14.215922 13995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:02:14.217087 13990 server_base.cc:1034] running on GCE node
I20260620 15:02:14.217808 13990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:02:14.218842 13990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:02:14.220077 13990 hybrid_clock.cc:648] HybridClock initialized: now 1781967734220031 us; error 33 us; skew 500 ppm
I20260620 15:02:14.220574 13990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:02:14 dist-test-slave-n5vv krb5kdc[13152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967734, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:02:14.223991 13990 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:02:14.225659 13990 webserver.cc:460] Webserver started at http://127.2.70.3:35779/ using document root <none> and password file <none>
I20260620 15:02:14.226388 13990 fs_manager.cc:359] Metadata directory not provided
I20260620 15:02:14.226531 13990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:02:14.226863 13990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:02:14 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967734, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:02:14.244853 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "f50060f859a441a9bd0d5efd5ef4e9d1"
format_stamp: "Formatted at 2026-06-20 15:02:14 on dist-test-slave-n5vv"
server_key: "cebcd35f9e2b74f0c5b75728bf715a14"
server_key_iv: "f1f0aa27c7ca911dc426b50f6088f1e6"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:14.245867 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "f50060f859a441a9bd0d5efd5ef4e9d1"
format_stamp: "Formatted at 2026-06-20 15:02:14 on dist-test-slave-n5vv"
server_key: "cebcd35f9e2b74f0c5b75728bf715a14"
server_key_iv: "f1f0aa27c7ca911dc426b50f6088f1e6"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:14.251750 13990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:02:14.267413 14005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:14.268867 13990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:02:14.269074 13990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "f50060f859a441a9bd0d5efd5ef4e9d1"
format_stamp: "Formatted at 2026-06-20 15:02:14 on dist-test-slave-n5vv"
server_key: "cebcd35f9e2b74f0c5b75728bf715a14"
server_key_iv: "f1f0aa27c7ca911dc426b50f6088f1e6"
server_key_version: "kuduclusterkey@0"
I20260620 15:02:14.269294 13990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:02:14.293220 13990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:02:14.297359 13990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:02:14.297709 13990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:02:14.298699 13990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:02:14.300529 13990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:02:14.300618 13990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:14.300783 13990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:02:14.300836 13990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:02:14.342551 13990 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:39117
I20260620 15:02:14.342643 14117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:39117 every 8 connection(s)
I20260620 15:02:14.343909 13990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:02:14.351200 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 13990
I20260620 15:02:14.351374 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:02:14.351851 2328 external_mini_cluster.cc:1428] Setting key e496f975b4015edaef9d7d02955b703e
Jun 20 15:02:14 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967734, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:02:14.363416 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.346497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:56353 (local address 127.2.70.62:46233)
0620 15:02:14.346945 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.346950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.347962 (+ 1012us) server_negotiation.cc:407] Connection header received
0620 15:02:14.349784 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.349789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.349897 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.350017 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:14.351457 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.352218 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.353850 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.354288 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.354423 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.358116 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:14.358121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:14.358124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:14.358157 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:14.360724 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.361408 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.361412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.361414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.361587 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.361981 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.361984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.361986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.362234 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:14.362348 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.363009 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.363162 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
I20260620 15:02:14.364339 14121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.347087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46233 (local address 127.2.70.3:56353)
0620 15:02:14.347802 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.347839 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:14.349447 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:14.350250 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:14.350259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:14.350677 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:02:14.351227 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.351244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.352359 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.352365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.353691 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.353711 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.354506 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.354513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.354748 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.355343 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:02:14.355364 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:02:14.357911 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:02:14.360848 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:14.360854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:14.360866 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:14.361295 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:14.361694 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:02:14.361697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:02:14.361699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:02:14.361884 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:02:14.362338 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:02:14.362343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:02:14.362692 (+ 349us) client_negotiation.cc:769] Sending connection context
0620 15:02:14.362915 (+ 223us) client_negotiation.cc:240] Negotiation successful
0620 15:02:14.363276 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":270,"threads_started":1}
I20260620 15:02:14.365773 14118 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46233
I20260620 15:02:14.366118 14118 heartbeater.cc:464] Registering TS with master...
I20260620 15:02:14.367007 14118 heartbeater.cc:511] Master 127.2.70.62:46233 requested a full tablet report, sending...
I20260620 15:02:14.368332 13628 ts_manager.cc:194] Registered new tserver with Master: f50060f859a441a9bd0d5efd5ef4e9d1 (127.2.70.3:39117)
I20260620 15:02:14.369148 13628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:56353
I20260620 15:02:14.379199 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:02:14.395723 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.383541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55046 (local address 127.2.70.62:46233)
0620 15:02:14.383711 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.383716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.384173 (+ 457us) server_negotiation.cc:407] Connection header received
0620 15:02:14.384305 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.384309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.384379 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.384461 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:14.385849 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.386532 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.387808 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.388203 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.388293 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.390968 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:14.390973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:14.390976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:14.391010 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:14.393462 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.394111 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.394114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.394116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.394227 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.394527 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.394530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.394532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.394790 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:14.394966 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.395305 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.395491 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:02:14.399669 13628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:02:14.406725 13628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:02:14.439284 14134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.419671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:39117 (local address 127.0.0.1:53432)
0620 15:02:14.427667 (+ 7996us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.427689 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:14.427835 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:14.430580 (+ 2745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:14.430588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:14.430616 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:02:14.431465 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.431478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.434268 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.434274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.436504 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.436519 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.438302 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.438310 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.438493 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.438546 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 15:02:14.438668 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 15:02:14.438860 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7840,"thread_start_us":6487,"threads_started":1}
I20260620 15:02:14.439464 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.420312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.2.70.3:39117)
0620 15:02:14.429992 (+ 9680us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.430000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.430034 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:02:14.430125 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.430132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.430371 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.430575 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:02:14.431681 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.434120 (+ 2439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.436664 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.438065 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.438228 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.438870 (+ 642us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.438942 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.439115 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9511,"thread_start_us":9346,"threads_started":1}
I20260620 15:02:14.439880 14131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.420115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.0.0.1:33990)
0620 15:02:14.428401 (+ 8286us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.428427 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:14.428549 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:14.432841 (+ 4292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:14.432846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:14.432869 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:02:14.433226 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.433238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.435287 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.435294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.437145 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.437159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.439048 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.439059 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.439224 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.439264 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 15:02:14.439386 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 15:02:14.439516 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8131,"thread_start_us":7259,"threads_started":1}
I20260620 15:02:14.440433 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.419986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.2.70.2:39003)
0620 15:02:14.431688 (+ 11702us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.431696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.431721 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:02:14.431810 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.431817 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.432092 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.432282 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:02:14.432938 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.434364 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.436494 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.438065 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.438208 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.438408 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.439987 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.440151 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11525,"thread_start_us":11392,"threads_started":1}
I20260620 15:02:14.439906 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.419530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:39003 (local address 127.0.0.1:36632)
0620 15:02:14.426632 (+ 7102us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.426690 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:14.426889 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:14.432281 (+ 5392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:14.432284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:14.432304 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:02:14.432726 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.432739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.434517 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.434527 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.436342 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.436353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.438967 (+ 2614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.438977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.439164 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.439202 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:02:14.439300 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:02:14.439425 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6786,"mutex_wait_us":28,"thread_start_us":8575,"threads_started":1}
I20260620 15:02:14.441082 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.420790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33990 (local address 127.2.70.1:46121)
0620 15:02:14.429182 (+ 8392us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.429190 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.429220 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:02:14.429327 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.429334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.432629 (+ 3295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.433424 (+ 795us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:02:14.433683 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.435139 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.437332 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.438779 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.440302 (+ 1523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.440490 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.440580 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.440732 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8239,"thread_start_us":8071,"threads_started":1}
I20260620 15:02:14.443849 14053 tablet_service.cc:1467] Processing CreateTablet for tablet 1715b20ef0fd44219407c133cf7b4922 (DEFAULT_TABLE table=test-table [id=090fa5f5fc4d4a67b7d22bf6450b19b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:02:14.444522 13919 tablet_service.cc:1467] Processing CreateTablet for tablet 1715b20ef0fd44219407c133cf7b4922 (DEFAULT_TABLE table=test-table [id=090fa5f5fc4d4a67b7d22bf6450b19b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:02:14.444521 13776 tablet_service.cc:1467] Processing CreateTablet for tablet 1715b20ef0fd44219407c133cf7b4922 (DEFAULT_TABLE table=test-table [id=090fa5f5fc4d4a67b7d22bf6450b19b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:02:14.445377 14053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1715b20ef0fd44219407c133cf7b4922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:14.445706 13919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1715b20ef0fd44219407c133cf7b4922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:14.445935 13776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1715b20ef0fd44219407c133cf7b4922. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:02:14.456925 14137 tablet_bootstrap.cc:492] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: Bootstrap starting.
I20260620 15:02:14.460175 14138 tablet_bootstrap.cc:492] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c: Bootstrap starting.
I20260620 15:02:14.460969 14139 tablet_bootstrap.cc:492] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Bootstrap starting.
I20260620 15:02:14.461944 14137 tablet_bootstrap.cc:654] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:14.463292 14137 log.cc:824] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:14.464417 14138 tablet_bootstrap.cc:654] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:14.464768 14139 tablet_bootstrap.cc:654] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Neither blocks nor log segments found. Creating new log.
I20260620 15:02:14.466293 14138 log.cc:824] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:14.466338 14139 log.cc:824] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Log is configured to *not* fsync() on all Append() calls
I20260620 15:02:14.469182 14137 tablet_bootstrap.cc:492] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: No bootstrap required, opened a new log
I20260620 15:02:14.469448 14137 ts_tablet_manager.cc:1397] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260620 15:02:14.471356 14139 tablet_bootstrap.cc:492] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: No bootstrap required, opened a new log
I20260620 15:02:14.471692 14139 ts_tablet_manager.cc:1397] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260620 15:02:14.473531 14138 tablet_bootstrap.cc:492] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c: No bootstrap required, opened a new log
I20260620 15:02:14.473796 14138 ts_tablet_manager.cc:1397] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20260620 15:02:14.480424 14137 raft_consensus.cc:348] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.480945 14137 raft_consensus.cc:374] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:14.481038 14137 raft_consensus.cc:729] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 354e12e61d32474daeff95115886c11d, State: Initialized, Role: FOLLOWER
I20260620 15:02:14.481850 14137 consensus_queue.cc:260] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.482018 14138 raft_consensus.cc:348] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.482434 14138 raft_consensus.cc:374] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:14.482551 14138 raft_consensus.cc:729] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaa1295ac6546638fa1917bd17baa9c, State: Initialized, Role: FOLLOWER
I20260620 15:02:14.482389 14139 raft_consensus.cc:348] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.482821 14139 raft_consensus.cc:374] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:02:14.482919 14139 raft_consensus.cc:729] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50060f859a441a9bd0d5efd5ef4e9d1, State: Initialized, Role: FOLLOWER
I20260620 15:02:14.483256 13984 heartbeater.cc:503] Master 127.2.70.62:46233 was elected leader, sending a full tablet report...
I20260620 15:02:14.483220 14138 consensus_queue.cc:260] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.483572 14139 consensus_queue.cc:260] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.483820 14137 ts_tablet_manager.cc:1428] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 15:02:14.485571 14118 heartbeater.cc:503] Master 127.2.70.62:46233 was elected leader, sending a full tablet report...
I20260620 15:02:14.486195 14139 ts_tablet_manager.cc:1428] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260620 15:02:14.487640 14138 ts_tablet_manager.cc:1428] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260620 15:02:14.545006 14143 raft_consensus.cc:482] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:02:14.545320 14143 raft_consensus.cc:504] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.547024 14143 leader_election.cc:288] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f50060f859a441a9bd0d5efd5ef4e9d1 (127.2.70.3:39117), 1eaa1295ac6546638fa1917bd17baa9c (127.2.70.1:46121)
I20260620 15:02:14.554793 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.547445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:47601 (local address 127.2.70.3:39117)
0620 15:02:14.547649 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.547655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.548443 (+ 788us) server_negotiation.cc:407] Connection header received
0620 15:02:14.548646 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.548651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.548730 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.548831 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:02:14.550245 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.551104 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.553015 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.553845 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.553988 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.554095 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.554396 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.554516 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:02:14.554806 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.547547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:39737 (local address 127.2.70.1:46121)
0620 15:02:14.547744 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.547752 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.548444 (+ 692us) server_negotiation.cc:407] Connection header received
0620 15:02:14.548646 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.548651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.548730 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.548831 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:02:14.550324 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.551288 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.553026 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.553856 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.554022 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.554126 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.554428 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.554554 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:02:14.554965 14147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.547365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:39117 (local address 127.2.70.2:47601)
0620 15:02:14.548293 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.548315 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:14.548511 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:14.548865 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:14.548870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:14.548915 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:02:14.549997 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.550012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.551244 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.551248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.552855 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.552868 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.554068 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.554075 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.554238 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.554265 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 15:02:14.554372 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 15:02:14.554500 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":162,"spinlock_wait_cycles":45952,"thread_start_us":661,"threads_started":1}
I20260620 15:02:14.555200 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.547384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.2.70.2:39737)
0620 15:02:14.548331 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.548349 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:02:14.548488 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:02:14.548865 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:02:14.548870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:02:14.548918 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:02:14.550116 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.550128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.551422 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.551431 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.552862 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:02:14.552875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.554088 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.554094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:02:14.554265 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.554287 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:02:14.554385 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:02:14.554513 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":85,"spinlock_wait_cycles":24320,"thread_start_us":462,"threads_started":1}
I20260620 15:02:14.555857 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1715b20ef0fd44219407c133cf7b4922" candidate_uuid: "354e12e61d32474daeff95115886c11d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" is_pre_election: true
I20260620 15:02:14.556205 13796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1715b20ef0fd44219407c133cf7b4922" candidate_uuid: "354e12e61d32474daeff95115886c11d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaa1295ac6546638fa1917bd17baa9c" is_pre_election: true
I20260620 15:02:14.556335 14073 raft_consensus.cc:2454] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 354e12e61d32474daeff95115886c11d in term 0.
I20260620 15:02:14.556690 13796 raft_consensus.cc:2454] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 354e12e61d32474daeff95115886c11d in term 0.
I20260620 15:02:14.557015 13873 leader_election.cc:302] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354e12e61d32474daeff95115886c11d, f50060f859a441a9bd0d5efd5ef4e9d1; no voters:
I20260620 15:02:14.557448 14143 raft_consensus.cc:2781] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:02:14.557660 14143 raft_consensus.cc:482] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:02:14.557789 14143 raft_consensus.cc:3037] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:14.560544 14143 raft_consensus.cc:504] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.561117 14143 leader_election.cc:288] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [CANDIDATE]: Term 1 election: Requested vote from peers f50060f859a441a9bd0d5efd5ef4e9d1 (127.2.70.3:39117), 1eaa1295ac6546638fa1917bd17baa9c (127.2.70.1:46121)
I20260620 15:02:14.561616 14073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1715b20ef0fd44219407c133cf7b4922" candidate_uuid: "354e12e61d32474daeff95115886c11d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1"
I20260620 15:02:14.561678 13796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1715b20ef0fd44219407c133cf7b4922" candidate_uuid: "354e12e61d32474daeff95115886c11d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaa1295ac6546638fa1917bd17baa9c"
I20260620 15:02:14.561829 14073 raft_consensus.cc:3037] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:14.561909 13796 raft_consensus.cc:3037] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:02:14.564054 14073 raft_consensus.cc:2454] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 354e12e61d32474daeff95115886c11d in term 1.
I20260620 15:02:14.564075 13796 raft_consensus.cc:2454] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 354e12e61d32474daeff95115886c11d in term 1.
I20260620 15:02:14.564435 13873 leader_election.cc:302] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 354e12e61d32474daeff95115886c11d, f50060f859a441a9bd0d5efd5ef4e9d1; no voters:
I20260620 15:02:14.564718 14143 raft_consensus.cc:2781] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:02:14.565165 14143 raft_consensus.cc:686] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 1 LEADER]: Becoming Leader. State: Replica: 354e12e61d32474daeff95115886c11d, State: Running, Role: LEADER
I20260620 15:02:14.565698 14143 consensus_queue.cc:237] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } }
I20260620 15:02:14.570063 13625 catalog_manager.cc:5495] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d reported cstate change: term changed from 0 to 1, leader changed from <none> to 354e12e61d32474daeff95115886c11d (127.2.70.2). New cstate: current_term: 1 leader_uuid: "354e12e61d32474daeff95115886c11d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "354e12e61d32474daeff95115886c11d" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 } health_report { overall_health: UNKNOWN } } }
W20260620 15:02:14.595543 14119 tablet.cc:2369] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:02:14.596091 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.589121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36638 (local address 127.2.70.2:39003)
0620 15:02:14.589347 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.589353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.589375 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:02:14.589504 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.589510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.589625 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.589783 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:02:14.590261 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.591726 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.593892 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.594281 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.594388 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.594900 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:02:14.595422 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:02:14.595548 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.595710 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.595800 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 15:02:14.604497 14073 raft_consensus.cc:1261] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 1 FOLLOWER]: Refusing update from remote peer 354e12e61d32474daeff95115886c11d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:02:14.604590 13796 raft_consensus.cc:1261] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 1 FOLLOWER]: Refusing update from remote peer 354e12e61d32474daeff95115886c11d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:02:14.605625 14143 consensus_queue.cc:1035] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaa1295ac6546638fa1917bd17baa9c" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:02:14.605947 14148 consensus_queue.cc:1035] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f50060f859a441a9bd0d5efd5ef4e9d1" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:02:14.622618 14152 mvcc.cc:205] Tried to move back new op lower bound from 7298939840930476032 to 7298939840785371136. Current Snapshot: MvccSnapshot[applied={T|T < 7298939840930476032}]
I20260620 15:02:14.631573 13625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:02:14.631860 13625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:02:14.635900 13625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a090ba2a3ce47509065e0ab0ba8ad71: Sending DeleteTablet for 3 replicas of tablet 1715b20ef0fd44219407c133cf7b4922
I20260620 15:02:14.636890 13919 tablet_service.cc:1514] Processing DeleteTablet for tablet 1715b20ef0fd44219407c133cf7b4922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:02:14 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:36632
I20260620 15:02:14.640825 14053 tablet_service.cc:1514] Processing DeleteTablet for tablet 1715b20ef0fd44219407c133cf7b4922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:02:14 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:53432
I20260620 15:02:14.641672 13776 tablet_service.cc:1514] Processing DeleteTablet for tablet 1715b20ef0fd44219407c133cf7b4922 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:02:14 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:33990
I20260620 15:02:14.647125 14161 tablet_replica.cc:330] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: stopping tablet replica
I20260620 15:02:14.647511 14161 raft_consensus.cc:2229] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:02:14.647985 14161 raft_consensus.cc:2258] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:02:14.654968 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 13708
I20260620 15:02:14.656067 14161 ts_tablet_manager.cc:1905] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:02:14.658474 14163 tablet_replica.cc:330] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c: stopping tablet replica
I20260620 15:02:14.663801 14162 tablet_replica.cc:330] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: stopping tablet replica
I20260620 15:02:14.665527 14162 raft_consensus.cc:2229] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:02:14.671396 14163 raft_consensus.cc:2229] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:02:14.672356 14163 raft_consensus.cc:2258] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:02:14.676234 14162 raft_consensus.cc:2258] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:02:14.679111 14162 ts_tablet_manager.cc:1905] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 20 15:02:14 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:02:14.686306 14162 ts_tablet_manager.cc:1918] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:02:14.686494 14162 log.cc:1196] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/1715b20ef0fd44219407c133cf7b4922
I20260620 15:02:14.686996 14162 ts_tablet_manager.cc:1939] T 1715b20ef0fd44219407c133cf7b4922 P f50060f859a441a9bd0d5efd5ef4e9d1: Deleting consensus metadata
I20260620 15:02:14.687336 14163 ts_tablet_manager.cc:1905] T 1715b20ef0fd44219407c133cf7b4922 P 1eaa1295ac6546638fa1917bd17baa9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:02:14.688808 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.656405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.2.70.1:46121)
0620 15:02:14.656589 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.656595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.667576 (+ 10981us) server_negotiation.cc:407] Connection header received
0620 15:02:14.667650 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.667655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.667740 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.667841 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:14.670019 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.671471 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.673599 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.674359 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.674453 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.680732 (+ 6279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:14.680748 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:14.680767 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:14.680815 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:14.684711 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.685650 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.685660 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.685664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.685844 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.686349 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.686354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.686357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.687183 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:14.687556 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.688164 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.688400 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:02:14.688899 13613 catalog_manager.cc:4841] TS f50060f859a441a9bd0d5efd5ef4e9d1 (127.2.70.3:39117): tablet 1715b20ef0fd44219407c133cf7b4922 (table test-table [id=090fa5f5fc4d4a67b7d22bf6450b19b1]) successfully deleted
I20260620 15:02:14.690510 13836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:02:14.693931 14161 ts_tablet_manager.cc:1918] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:02:14.694128 14161 log.cc:1196] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/1715b20ef0fd44219407c133cf7b4922
I20260620 15:02:14.694685 14161 ts_tablet_manager.cc:1939] T 1715b20ef0fd44219407c133cf7b4922 P 354e12e61d32474daeff95115886c11d: Deleting consensus metadata
I20260620 15:02:14.696519 13612 catalog_manager.cc:4841] TS 354e12e61d32474daeff95115886c11d (127.2.70.2:39003): tablet 1715b20ef0fd44219407c133cf7b4922 (table test-table [id=090fa5f5fc4d4a67b7d22bf6450b19b1]) successfully deleted
I20260620 15:02:14.811887 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 13708
I20260620 15:02:14.830598 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 13847
W20260620 15:02:14.830617 13615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.1:46121 (error 108)
I20260620 15:02:14.832646 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.832159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.0.0.1:34018)
0620 15:02:14.832332 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.832478 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 15:02:14.834038 13615 catalog_manager.cc:4584] TS 1eaa1295ac6546638fa1917bd17baa9c (127.2.70.1:46121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1715b20ef0fd44219407c133cf7b4922: Network error: Client connection negotiation failed: client connection to 127.2.70.1:46121: connect: Connection refused (error 111)
Jun 20 15:02:14 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:02:14.848428 14133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.831777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.2.70.2:39003)
0620 15:02:14.831958 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.831965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.831984 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:02:14.832048 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.832054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.832129 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.832544 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:14.833979 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.835218 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.836480 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.837127 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.837251 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.841447 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:14.841464 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:14.841503 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:14.841549 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:14.845664 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.846311 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.846322 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.846331 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.846500 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.846878 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.846883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.846887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.847317 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:14.847639 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.847824 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.848033 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:02:14.849171 13979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:02:14.887590 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.887057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.0.0.1:34034)
0620 15:02:14.887284 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.887388 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:02:14.888652 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.888273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.0.0.1:34042)
0620 15:02:14.888426 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.888524 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 15:02:14.949435 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.948898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.0.0.1:34048)
0620 15:02:14.949138 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.949248 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 15:02:14.950480 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.950127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.0.0.1:34054)
0620 15:02:14.950278 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:14.950356 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:02:14.966285 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 13847
I20260620 15:02:14.984514 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 13990
Jun 20 15:02:14 dist-test-slave-n5vv krb5kdc[13152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967730, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:02:14.998579 14135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:14.985266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.2.70.3:39117)
0620 15:02:14.985426 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:02:14.985431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:02:14.985450 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:02:14.985692 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:02:14.985696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:02:14.985760 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:02:14.985846 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:02:14.987130 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.987806 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.989101 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:02:14.989465 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:02:14.989618 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:02:14.993626 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:02:14.993639 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:02:14.993655 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:02:14.993690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:02:14.996154 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.996756 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.996763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.996765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.996894 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:02:14.997207 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:02:14.997210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:02:14.997212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:02:14.997641 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:02:14.997872 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 15:02:14.998155 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 15:02:14.998320 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:02:14.999194 14113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:02:15.031157 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:15.030559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.0.0.1:34068)
0620 15:02:15.030792 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:15.030912 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 15:02:15.032234 14132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:02:15.031803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:46121 (local address 127.0.0.1:34074)
0620 15:02:15.032023 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:02:15.032107 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260620 15:02:15.088395 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 13990
I20260620 15:02:15.104753 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 13596
I20260620 15:02:15.105962 13658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:02:15.218379 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 13596
I20260620 15:02:15.239852 2328 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:02:15.575187 2328 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:02:15.575394 2328 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:02:15.920873 2328 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:02:15.921 UTC [13182] LOG: received smart shutdown request
2026-06-20 15:02:15.924 UTC [13182] LOG: background worker "logical replication launcher" (PID 13191) exited with exit code 1
2026-06-20 15:02:15.924 UTC [13186] LOG: shutting down
2026-06-20 15:02:16.096 UTC [13182] LOG: database system is shut down
2026-06-20T15:02:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49140 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:02:16 dist-test-slave-n5vv krb5kdc[14178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:02:16 dist-test-slave-n5vv krb5kdc[14178](info): set up 2 sockets
Jun 20 15:02:16 dist-test-slave-n5vv krb5kdc[14178](info): commencing operation
krb5kdc: starting...
W20260620 15:02:17.353796 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:02:17 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967737, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:02:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.70.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.70.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/rangeradmin_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/rangeradmin_127.2.70.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.70.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.70.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/rangerlookup_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/rangerlookup_127.2.70.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.70.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.70.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/HTTP_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/HTTP_127.2.70.20@KRBTEST.COM.keytab.
I20260620 15:02:17.681329 2328 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskK9gJB7/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/postgres -l logfile start
2026-06-20 15:02:20.202 UTC [14208] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 15:02:20.203 UTC [14208] LOG: listening on IPv4 address "127.2.70.20", port 43513
2026-06-20 15:02:20.207 UTC [14208] LOG: listening on Unix socket "/tmp/.s.PGSQL.43513"
2026-06-20 15:02:20.227 UTC [14211] LOG: database system was shut down at 2026-06-20 15:02:18 UTC
2026-06-20 15:02:20.231 UTC [14208] LOG: database system is ready to accept connections
I20260620 15:02:21.221581 2328 mini_postgres.cc:96] Postgres bound to 43513
2026-06-20 15:02:21.238 UTC [14219] FATAL: database "slave" does not exist
127.2.70.20:43513 - accepting connections
I20260620 15:02:21.239125 2328 mini_ranger.cc:161] Starting Ranger...
I20260620 15:02:21.264192 2328 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 15:02:21.740825 2328 mini_ranger.cc:87] Created ranger Postgres database
I20260620 15:02:21.740942 2328 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-admin
2026-06-20 15:02:22,019 [I] DB FLAVOR :POSTGRES
2026-06-20 15:02:22,019 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:02:22,019 [I] Checking connection..
2026-06-20 15:02:22,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 15:02:22,264 [I] Checking connection passed.
2026-06-20 15:02:22,265 [I] --------- Verifying version history table ---------
2026-06-20 15:02:22,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:02:22,509 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:02:22,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:02:22,774 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 15:02:22,775 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 15:02:22,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 15:02:23.027 UTC [14283] WARNING: there is no transaction in progress
2026-06-20 15:02:23,033 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 15:02:23,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 15:02:23,274 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 15:02:23,274 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:02:23,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 15:02:23,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 15:02:23,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n5vv.c.gcp-upstream.internal','N') ;"
2026-06-20 15:02:24,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:02:24,235 [I] Table x_portal_user does not exist in database ranger
2026-06-20 15:02:24,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:02:24,479 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 15:02:24,479 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:02:24,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:02:24,980 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 15:02:24,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 15:02:25.920 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:25.937 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:25.954 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:25.970 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:25.987 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.005 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.059 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.066 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.079 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.089 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.100 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.114 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.123 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.129 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.136 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26.142 UTC [14391] WARNING: there is no transaction in progress
2026-06-20 15:02:26,751 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 15:02:26,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 15:02:26,984 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 15:02:26,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 15:02:27,234 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 15:02:27,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 15:02:27,480 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 15:02:27,481 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 15:02:27,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 15:02:27,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n5vv.c.gcp-upstream.internal';"
2026-06-20 15:02:28,243 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 15:02:28,244 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 15:02:28,501 [I] DB_PATCHES have already been applied
I20260620 15:02:28.508270 2328 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-admin:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskK9gJB7/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 15:02:28.508389 2328 mini_ranger.cc:193] Using host: 127.2.70.20
I20260620 15:02:28.524933 2328 mini_ranger.cc:227] Ranger admin URL: http://127.2.70.20:57649
Jun 20, 2026 3:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 3:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 3:02:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 3:02:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 3:02:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 15:02:29.677460 2328 mini_ranger.cc:160] Time spent starting Ranger: real 8.438s user 0.003s sys 0.036s
Jun 20, 2026 3:02:50 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 3:02:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 3:02:50 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 3:02:50 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 3:02:51 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 15:02:51.756387 2328 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.70.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.70.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/rangerkms_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.70.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/rangerkms_127.2.70.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.70.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.70.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.70.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/HTTP_127.2.70.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.70.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/HTTP_127.2.70.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 15:02:51 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967771, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 15:02:51.929354 2328 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 15:02:51.952173 2328 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 15:02:52.778301 2328 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 15:02:52.778425 2328 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms
I20260620 15:02:52.778465 2328 mini_ranger_kms.cc:227] Using postgres at 127.2.70.20:43513
2026-06-20 15:02:53,156 [I] DB FLAVOR :POSTGRES
2026-06-20 15:02:53,156 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 15:02:53,156 [I] Checking connection
2026-06-20 15:02:53,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 15:02:53,413 [I] connection success
2026-06-20 15:02:53,413 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 15:02:53,413 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 15:02:53,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 15:02:53,681 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 15:02:53,682 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 15:02:53,682 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 15:02:53,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.70.20:43513/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 15:02:53,990 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 15:02:54.474583 2328 mini_ranger_kms.cc:310] Created kms service
I20260620 15:02:54.682503 2328 mini_ranger_kms.cc:326] Created kudu user
I20260620 15:02:54.729810 2328 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 15:02:54 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967771, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:02:55.129725 2328 mini_ranger_kms.cc:384] Added ranger policy
I20260620 15:02:55.130087 2328 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms:/tmp/dist-test-taskK9gJB7/build/asan/bin/postgresql.jar:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskK9gJB7/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 15:02:55.130159 2328 mini_ranger_kms.cc:240] Using host: 127.2.70.20
I20260620 15:02:55.147101 2328 mini_ranger_kms.cc:276] Ranger KMS PID: 14571
I20260620 15:02:55.147228 2328 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.70.20:49563
Jun 20, 2026 3:02:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 3:02:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 3:02:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 3:02:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 3:02:56 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 3:02:56 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 15:02:57.512167 2328 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.583s user 0.061s sys 0.339s
I20260620 15:02:57.512416 2328 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 15:02:57.512599 2328 mini_ranger_kms.cc:401] 127.2.70.20:49563/kms/v1/keys
Jun 20 15:02:57 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967777, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 15:02:58 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967777, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.20@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
Jun 20 15:02:58 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967777, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.70.20@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
Jun 20 15:03:00 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:03:00.873448 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45193
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:45193
--ranger_config_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:49563/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:01.207612 14622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:01.207967 14622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:01.208127 14622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:01.215605 14622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:01.215761 14622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:01.215860 14622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:01.215960 14622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:01.216042 14622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:01.226555 14622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.70.20:49563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:45193
--ranger_config_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:01.227941 14622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:01.229306 14622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:01.241791 14630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:01.241863 14627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:01.241839 14628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:01.242458 14622 server_base.cc:1034] running on GCE node
I20260620 15:03:01.243279 14622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:01.244669 14622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:01.245870 14622 hybrid_clock.cc:648] HybridClock initialized: now 1781967781245809 us; error 52 us; skew 500 ppm
I20260620 15:03:01.246201 14622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:01 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967781, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:01.249240 14622 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:03:01.250741 14622 webserver.cc:460] Webserver started at http://127.2.70.62:32775/ using document root <none> and password file <none>
I20260620 15:03:01.251353 14622 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:01.251446 14622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:01.251704 14622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:03:01 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967781, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:01.319872 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "243b0a74afdc45ab8aa87c629dc75eb5"
format_stamp: "Formatted at 2026-06-20 15:03:01 on dist-test-slave-n5vv"
server_key: "744af1321a4a412ebe0c1b8cce0e5d92"
server_key_iv: "837971e5f117e406093b68c6f1c8a70d"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:01.321071 14622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "243b0a74afdc45ab8aa87c629dc75eb5"
format_stamp: "Formatted at 2026-06-20 15:03:01 on dist-test-slave-n5vv"
server_key: "744af1321a4a412ebe0c1b8cce0e5d92"
server_key_iv: "837971e5f117e406093b68c6f1c8a70d"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:01.328660 14622 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260620 15:03:01.346541 14637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:01.348165 14622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:01.348380 14622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "243b0a74afdc45ab8aa87c629dc75eb5"
format_stamp: "Formatted at 2026-06-20 15:03:01 on dist-test-slave-n5vv"
server_key: "744af1321a4a412ebe0c1b8cce0e5d92"
server_key_iv: "837971e5f117e406093b68c6f1c8a70d"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:01.348629 14622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:01.369086 14622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:01.373584 14622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:01.373970 14622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:01.394454 14622 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:45193
I20260620 15:03:01.394471 14688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:45193 every 8 connection(s)
I20260620 15:03:01.395938 14622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:03:01.400036 14689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:01.400629 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 14622
I20260620 15:03:01.400844 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:03:01.401607 2328 external_mini_cluster.cc:1428] Setting key 5e60db1830606b04942631a6e42477b8
I20260620 15:03:01.414659 14689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Bootstrap starting.
Jun 20 15:03:01 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:01.418361 14689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:01.419767 14689 log.cc:824] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:01.423380 14689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: No bootstrap required, opened a new log
I20260620 15:03:01.426784 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:01.403744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.2.70.62:45193)
0620 15:03:01.404642 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:01.404659 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:01.404709 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:03:01.406912 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:01.406944 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:01.407503 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:01.407995 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:01.409278 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:01.411341 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:01.412730 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:01.413841 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:01.414016 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:01.418317 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:01.418331 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:01.418357 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:01.418402 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:01.422209 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:01.422919 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:01.422927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:01.422935 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:01.423134 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:01.423494 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:01.423499 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:01.423502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:01.424245 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:01.424524 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:01.424920 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 15:03:01.425341 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":413,"threads_started":1}
I20260620 15:03:01.432888 14689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } }
I20260620 15:03:01.433231 14689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:01.433310 14689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243b0a74afdc45ab8aa87c629dc75eb5, State: Initialized, Role: FOLLOWER
I20260620 15:03:01.433859 14689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } }
I20260620 15:03:01.434060 14689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:03:01.434154 14689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:03:01.434275 14689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:01.436379 14689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } }
I20260620 15:03:01.436818 14689 leader_election.cc:302] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 243b0a74afdc45ab8aa87c629dc75eb5; no voters:
I20260620 15:03:01.437566 14689 leader_election.cc:288] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:03:01.437726 14694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:01.438369 14694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 1 LEADER]: Becoming Leader. State: Replica: 243b0a74afdc45ab8aa87c629dc75eb5, State: Running, Role: LEADER
I20260620 15:03:01.438995 14694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } }
I20260620 15:03:01.440153 14689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:03:01.441886 14696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } } }
I20260620 15:03:01.441929 14695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 243b0a74afdc45ab8aa87c629dc75eb5. Latest consensus state: current_term: 1 leader_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } } }
I20260620 15:03:01.442348 14695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:01.442243 14696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:01.446714 14699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:03:01.452737 14699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:03:01.453831 14689 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 15:03:01.472123 14699 catalog_manager.cc:1347] Generated new cluster ID: ed3000ba4d4f4989bca1455a31d7256a
I20260620 15:03:01.472290 14699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:03:01.493364 14699 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:03:01.495050 14699 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:03:01.504279 14699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Generated new TSK 0
I20260620 15:03:01.505553 14699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:03:02.259682 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45193
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:03:02 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 15:03:02.712796 14734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:02.713215 14734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:02.713421 14734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:02.725203 14734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:02.725351 14734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:02.725636 14734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:03:02.741613 14734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.70.20:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:02.743523 14734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:02.745260 14734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:02.767849 14744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:02.769613 14745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:02.778095 14734 server_base.cc:1034] running on GCE node
W20260620 15:03:02.782161 14747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:02.782912 14734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:02.789811 14734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:02.791069 14734 hybrid_clock.cc:648] HybridClock initialized: now 1781967782790993 us; error 78 us; skew 500 ppm
I20260620 15:03:02.791432 14734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:02 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:02.794909 14734 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:03:02.796761 14734 webserver.cc:460] Webserver started at http://127.2.70.1:44269/ using document root <none> and password file <none>
I20260620 15:03:02.797570 14734 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:02.797696 14734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:02.798022 14734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:03:02 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:02.820339 14734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "e5c8164242dc4de1b9b76c727de464de"
format_stamp: "Formatted at 2026-06-20 15:03:02 on dist-test-slave-n5vv"
server_key: "602318fd103fc57b304ccc5141f1860d"
server_key_iv: "5bebfec13408031cd9ecd01dcf60556c"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:02.821736 14734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "e5c8164242dc4de1b9b76c727de464de"
format_stamp: "Formatted at 2026-06-20 15:03:02 on dist-test-slave-n5vv"
server_key: "602318fd103fc57b304ccc5141f1860d"
server_key_iv: "5bebfec13408031cd9ecd01dcf60556c"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:02.835634 14734 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.008s
I20260620 15:03:02.853757 14754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:02.855391 14734 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:03:02.855589 14734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "e5c8164242dc4de1b9b76c727de464de"
format_stamp: "Formatted at 2026-06-20 15:03:02 on dist-test-slave-n5vv"
server_key: "602318fd103fc57b304ccc5141f1860d"
server_key_iv: "5bebfec13408031cd9ecd01dcf60556c"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:02.855839 14734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:02.911674 14734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:02.917330 14734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:02.917711 14734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:02.918839 14734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:02.920742 14734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:02.920845 14734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:02.920996 14734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:02.921085 14734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:03:02 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:03.026124 14866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:43157 every 8 connection(s)
I20260620 15:03:03.026489 14734 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:43157
I20260620 15:03:03.028281 14734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:03:03.036823 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 14734
I20260620 15:03:03.037074 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:03:03.038015 2328 external_mini_cluster.cc:1428] Setting key 4a0932d73a15ef511a66e67b6bdbac27
Jun 20 15:03:03 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
I20260620 15:03:03.096589 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:03.036623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45193 (local address 127.2.70.1:38413)
0620 15:03:03.043941 (+ 7318us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:03.044008 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:03.046395 (+ 2387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:03.062202 (+ 15807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:03.062217 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:03.062965 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:03.063856 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:03.063884 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:03.065776 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:03.065783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:03.067971 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:03.067987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:03.069677 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:03.069684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:03.069971 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:03.070803 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:03.070832 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:03.077867 (+ 7035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:03.083201 (+ 5334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:03.083212 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:03.083232 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:03.083827 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:03.084424 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:03.084429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:03.084433 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:03.084763 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:03.093768 (+ 9005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:03.093778 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:03.094265 (+ 487us) client_negotiation.cc:769] Sending connection context
0620 15:03:03.094585 (+ 320us) client_negotiation.cc:240] Negotiation successful
0620 15:03:03.095178 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4683,"thread_start_us":8740,"threads_started":1}
I20260620 15:03:03.098351 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:03.037433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:38413 (local address 127.2.70.62:45193)
0620 15:03:03.061201 (+ 23768us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:03.061208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:03.061231 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:03:03.061303 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:03.061308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:03.061424 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:03.064160 (+ 2736us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:03.064335 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:03.065469 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:03.068194 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:03.068825 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:03.078249 (+ 9424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:03.078787 (+ 538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:03.078794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:03.078799 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:03.078832 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:03.082847 (+ 4015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:03.083996 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:03.084001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:03.084005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:03.084282 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:03.093124 (+ 8842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:03.093130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:03.093134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:03.093563 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:03.097584 (+ 4021us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:03.097734 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 15:03:03.097975 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23624,"thread_start_us":24116,"threads_started":1}
I20260620 15:03:03.099893 14867 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45193
I20260620 15:03:03.100217 14867 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:03.101011 14867 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
I20260620 15:03:03.103549 14654 ts_manager.cc:194] Registered new tserver with Master: e5c8164242dc4de1b9b76c727de464de (127.2.70.1:43157)
I20260620 15:03:03.105631 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:38413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:03:03.124483 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45193
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:03:03 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967782, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
W20260620 15:03:03.584527 14873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:03.585032 14873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:03.585230 14873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:03.597221 14873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:03.597500 14873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:03.597757 14873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:03:03.613842 14873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.70.20:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:03.615510 14873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:03.617213 14873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:03.630972 14890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:03.631016 14888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:03.630910 14887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:03.632328 14873 server_base.cc:1034] running on GCE node
I20260620 15:03:03.632974 14873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:03.634131 14873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:03.635353 14873 hybrid_clock.cc:648] HybridClock initialized: now 1781967783635306 us; error 42 us; skew 500 ppm
I20260620 15:03:03.635711 14873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:03 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:03.638809 14873 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:03:03.640374 14873 webserver.cc:460] Webserver started at http://127.2.70.2:44353/ using document root <none> and password file <none>
I20260620 15:03:03.641131 14873 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:03.641274 14873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:03.641664 14873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:03:03 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:03.658493 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "fd8982c3eef941cca60f8a5e1fd47cd5"
format_stamp: "Formatted at 2026-06-20 15:03:03 on dist-test-slave-n5vv"
server_key: "34c868afc621b4a1b98f4c5999a9cacb"
server_key_iv: "fbe85815210d973c68b99cc85e988b40"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:03.659693 14873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "fd8982c3eef941cca60f8a5e1fd47cd5"
format_stamp: "Formatted at 2026-06-20 15:03:03 on dist-test-slave-n5vv"
server_key: "34c868afc621b4a1b98f4c5999a9cacb"
server_key_iv: "fbe85815210d973c68b99cc85e988b40"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:03.665175 14873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:03:03.679296 14897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:03.681104 14873 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:03:03.681295 14873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "fd8982c3eef941cca60f8a5e1fd47cd5"
format_stamp: "Formatted at 2026-06-20 15:03:03 on dist-test-slave-n5vv"
server_key: "34c868afc621b4a1b98f4c5999a9cacb"
server_key_iv: "fbe85815210d973c68b99cc85e988b40"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:03.681565 14873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:03.696686 14873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:03.700932 14873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:03.701244 14873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:03.702276 14873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:03.703639 14873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:03.703742 14873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:03.703886 14873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:03.703970 14873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:03.736204 14873 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:38243
I20260620 15:03:03.736238 15009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:38243 every 8 connection(s)
I20260620 15:03:03.737540 14873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:03:03.741201 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 14873
I20260620 15:03:03.741415 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:03:03.741955 2328 external_mini_cluster.cc:1428] Setting key 1ee24285ec0b9e8b93a56673b383e0e1
Jun 20 15:03:03 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967783, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:03.759332 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:03.740062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:53373 (local address 127.2.70.62:45193)
0620 15:03:03.740493 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:03.740497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:03.741731 (+ 1234us) server_negotiation.cc:407] Connection header received
0620 15:03:03.743558 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:03.743563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:03.743648 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:03.743764 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:03.745207 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:03.746025 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:03.747603 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:03.748077 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:03.748214 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:03.751980 (+ 3766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:03.751989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:03.751997 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:03.752044 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:03.755778 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:03.756516 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:03.756524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:03.756530 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:03.756693 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:03.757112 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:03.757120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:03.757127 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:03.757557 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:03.757702 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:03.758589 (+ 887us) server_negotiation.cc:299] Negotiation successful
0620 15:03:03.758840 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":225,"threads_started":1}
I20260620 15:03:03.760118 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:03.740658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45193 (local address 127.2.70.2:53373)
0620 15:03:03.741552 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:03.741603 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:03.743264 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:03.744071 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:03.744080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:03.744489 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:03.745045 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:03.745062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:03.746181 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:03.746187 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:03.747460 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:03.747472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:03.748317 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:03.748325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:03.748564 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:03.749258 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:03.749279 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:03.751698 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:03.755929 (+ 4231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:03.755937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:03.755950 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:03.756393 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:03.756801 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:03.756804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:03.756806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:03.756999 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:03.757727 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:03.757739 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:03.758212 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 15:03:03.758450 (+ 238us) client_negotiation.cc:240] Negotiation successful
0620 15:03:03.758954 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":413,"threads_started":1}
I20260620 15:03:03.761822 15010 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45193
I20260620 15:03:03.762212 15010 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:03.763155 15010 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
I20260620 15:03:03.764786 14654 ts_manager.cc:194] Registered new tserver with Master: fd8982c3eef941cca60f8a5e1fd47cd5 (127.2.70.2:38243)
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:03:03.765753 14654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:53373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:03:03.793398 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45193
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:03:04.109426 14867 heartbeater.cc:503] Master 127.2.70.62:45193 was elected leader, sending a full tablet report...
W20260620 15:03:04.117439 15016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:04.117795 15016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:04.117959 15016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:04.125265 15016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:04.125420 15016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:04.125635 15016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:03:04.135439 15016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.70.20:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:04.136746 15016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:04.138149 15016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:04.151584 15022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:04.151734 15021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:04.152213 15016 server_base.cc:1034] running on GCE node
W20260620 15:03:04.152249 15024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:04.152850 15016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:04.153887 15016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:04.155115 15016 hybrid_clock.cc:648] HybridClock initialized: now 1781967784155073 us; error 28 us; skew 500 ppm
I20260620 15:03:04.155422 15016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:04 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967784, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:04.158463 15016 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:03:04.159825 15016 webserver.cc:460] Webserver started at http://127.2.70.3:45329/ using document root <none> and password file <none>
I20260620 15:03:04.160580 15016 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:04.160737 15016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:04.161034 15016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 15:03:04 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967784, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:04.175189 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "246f025a3d2d45d592d720ce32897a42"
format_stamp: "Formatted at 2026-06-20 15:03:04 on dist-test-slave-n5vv"
server_key: "5a52cbb6163e097972ec4516dea363a4"
server_key_iv: "00a4a107f7039729652d21e080e370f4"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:04.176079 15016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "246f025a3d2d45d592d720ce32897a42"
format_stamp: "Formatted at 2026-06-20 15:03:04 on dist-test-slave-n5vv"
server_key: "5a52cbb6163e097972ec4516dea363a4"
server_key_iv: "00a4a107f7039729652d21e080e370f4"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:04.181722 15016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:03:04.198335 15031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:04.199910 15016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:03:04.200114 15016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "246f025a3d2d45d592d720ce32897a42"
format_stamp: "Formatted at 2026-06-20 15:03:04 on dist-test-slave-n5vv"
server_key: "5a52cbb6163e097972ec4516dea363a4"
server_key_iv: "00a4a107f7039729652d21e080e370f4"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:04.200344 15016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:04.217448 15016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:04.220993 15016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:04.221306 15016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:04.222285 15016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:04.223718 15016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:04.223821 15016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:04.223973 15016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:04.224051 15016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:04.257606 15016 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:44067
I20260620 15:03:04.257697 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:44067 every 8 connection(s)
I20260620 15:03:04.258950 15016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:03:04.266546 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 15016
I20260620 15:03:04.266736 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:03:04.267412 2328 external_mini_cluster.cc:1428] Setting key 7078e19c3c14235358c66f3cf489498e
Jun 20 15:03:04 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967784, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:04.280093 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:04.261704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:57715 (local address 127.2.70.62:45193)
0620 15:03:04.262255 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:04.262267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:04.263236 (+ 969us) server_negotiation.cc:407] Connection header received
0620 15:03:04.265189 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:04.265195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:04.265316 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:04.265461 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:04.267434 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.268332 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.270415 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.270841 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.270937 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:04.274406 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:04.274411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:04.274414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:04.274446 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:04.277150 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:04.277995 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:04.277999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:04.278001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:04.278122 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:04.278605 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:04.278608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:04.278610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:04.278861 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:04.278976 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:04.279687 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 15:03:04.279847 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":304,"threads_started":1}
I20260620 15:03:04.281395 15147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:04.262317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45193 (local address 127.2.70.3:57715)
0620 15:03:04.263071 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:04.263115 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:04.264896 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:04.265742 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:04.265751 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:04.266327 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:04.267238 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:04.267266 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.268547 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.268552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:04.270281 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:04.270294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.271040 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.271045 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:04.271255 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:04.271978 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:04.272001 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:04.274188 (+ 2187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:04.277289 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:04.277296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:04.277310 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:04.277881 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:04.278222 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:04.278226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:04.278228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:04.278498 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:04.278975 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:04.278988 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:04.279304 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 15:03:04.279603 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 15:03:04.280001 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":312,"threads_started":1}
I20260620 15:03:04.283020 15144 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45193
I20260620 15:03:04.283357 15144 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:04.284065 15144 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
I20260620 15:03:04.285610 14653 ts_manager.cc:194] Registered new tserver with Master: 246f025a3d2d45d592d720ce32897a42 (127.2.70.3:44067)
I20260620 15:03:04.286612 14653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:57715
I20260620 15:03:04.295462 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:03:04.297124 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 14734
Jun 20 15:03:04 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:03:04.313632 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:04.298088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57958 (local address 127.2.70.1:43157)
0620 15:03:04.298707 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:04.298714 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:04.298739 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:03:04.298795 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:04.298801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:04.299008 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:04.299142 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:04.300554 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.301654 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.303162 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.303756 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.303855 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:04.308256 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:04.308268 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:04.308282 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:04.308319 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:04.311237 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:04.311884 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:04.311891 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:04.311893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:04.312045 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:04.312337 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:04.312340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:04.312342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:04.312648 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:04.312869 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:04.313089 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 15:03:04.313285 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":309,"threads_started":1}
I20260620 15:03:04.314896 14862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:04.397432 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 14734
I20260620 15:03:04.413677 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 14873
Jun 20 15:03:04 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:03:04.431121 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:04.414612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.2.70.2:38243)
0620 15:03:04.415200 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:04.415207 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:04.415233 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:03:04.415293 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:04.415300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:04.415530 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:04.415658 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:04.416770 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.417834 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.419082 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.419621 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.419718 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:04.424133 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:04.424145 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:04.424160 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:04.424204 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:04.427761 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:04.428610 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:04.428618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:04.428621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:04.428781 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:04.429217 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:04.429222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:04.429226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:04.429691 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:04.430029 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:04.430404 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 15:03:04.430726 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":306,"threads_started":1}
I20260620 15:03:04.432585 15005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:04.519299 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 14873
I20260620 15:03:04.536116 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 15016
Jun 20 15:03:04 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:03:04.552392 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:04.537100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.2.70.3:44067)
0620 15:03:04.537750 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:04.537757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:04.537786 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:03:04.537844 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:04.537851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:04.538065 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:04.538205 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:04.539373 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.540394 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.541887 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:04.542433 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:04.542542 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:04.546808 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:04.546819 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:04.546834 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:04.546872 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:04.549928 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:04.550577 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:04.550584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:04.550587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:04.550709 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:04.551014 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:04.551017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:04.551019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:04.551336 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:04.551579 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:04.551879 (+ 300us) server_negotiation.cc:299] Negotiation successful
0620 15:03:04.552116 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":291,"threads_started":1}
I20260620 15:03:04.553710 15139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:04.637794 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 15016
I20260620 15:03:04.668946 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 14622
I20260620 15:03:04.670362 14684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:04.777917 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 14622
I20260620 15:03:04.800987 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45193
--webserver_interface=127.2.70.62
--webserver_port=32775
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:45193
--ranger_config_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:49563/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.70.20:49563/kms with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:05.132308 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:05.132608 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:05.132697 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:05.140087 15156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:05.140204 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:05.140250 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:05.140306 15156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:05.140353 15156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:05.150440 15156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.70.20:49563/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:45193
--ranger_config_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=32775
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:05.151723 15156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:05.153024 15156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:05.164844 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:05.164779 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:05.165251 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:05.165511 15156 server_base.cc:1034] running on GCE node
I20260620 15:03:05.166376 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:05.167793 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:05.169023 15156 hybrid_clock.cc:648] HybridClock initialized: now 1781967785168978 us; error 59 us; skew 500 ppm
I20260620 15:03:05.169368 15156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:05 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967785, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:05.172523 15156 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:03:05.174003 15156 webserver.cc:460] Webserver started at http://127.2.70.62:32775/ using document root <none> and password file <none>
I20260620 15:03:05.174666 15156 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:05.174814 15156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:03:05 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967785, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:05.228508 15156 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:03:05.231734 15172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:05.233152 15156 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:03:05.233377 15156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "243b0a74afdc45ab8aa87c629dc75eb5"
format_stamp: "Formatted at 2026-06-20 15:03:01 on dist-test-slave-n5vv"
server_key: "744af1321a4a412ebe0c1b8cce0e5d92"
server_key_iv: "837971e5f117e406093b68c6f1c8a70d"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:05.234434 15156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:05.246196 15156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:05.249667 15156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:05.249899 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:05.267933 15156 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:45193
I20260620 15:03:05.267941 15223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:45193 every 8 connection(s)
I20260620 15:03:05.269377 15156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:03:05.274360 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 15156
I20260620 15:03:05.275223 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:43157
--local_ip_for_outbound_sockets=127.2.70.1
--tserver_master_addrs=127.2.70.62:45193
--webserver_port=44269
--webserver_interface=127.2.70.1
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.70.20:49563/kms with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:03:05.276232 15224 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:03:05.279489 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Bootstrap starting.
I20260620 15:03:05.305315 15224 log.cc:824] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:05.316509 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:03:05.317131 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Bootstrap complete.
I20260620 15:03:05.330149 15224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } }
I20260620 15:03:05.331425 15224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 243b0a74afdc45ab8aa87c629dc75eb5, State: Initialized, Role: FOLLOWER
I20260620 15:03:05.332162 15224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } }
I20260620 15:03:05.332444 15224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:03:05.332580 15224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:03:05.332768 15224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:03:05.338523 15224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } }
I20260620 15:03:05.339253 15224 leader_election.cc:302] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 243b0a74afdc45ab8aa87c629dc75eb5; no voters:
I20260620 15:03:05.339962 15224 leader_election.cc:288] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:03:05.340116 15228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:03:05.341648 15228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [term 2 LEADER]: Becoming Leader. State: Replica: 243b0a74afdc45ab8aa87c629dc75eb5, State: Running, Role: LEADER
I20260620 15:03:05.342233 15228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } }
I20260620 15:03:05.342581 15224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:03:05.345078 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 243b0a74afdc45ab8aa87c629dc75eb5. Latest consensus state: current_term: 2 leader_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } } }
I20260620 15:03:05.345863 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:05.345037 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "243b0a74afdc45ab8aa87c629dc75eb5" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45193 } } }
I20260620 15:03:05.346920 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:05.347888 15235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:03:05.361436 15235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:03:05.362949 15235 catalog_manager.cc:1259] Loaded cluster ID: ed3000ba4d4f4989bca1455a31d7256a
I20260620 15:03:05.363128 15235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:03:05.367121 15235 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:03:05.368610 15235 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Loaded TSK: 0
I20260620 15:03:05.370270 15235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:03:05.372795 15224 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260620 15:03:05.739562 15226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:05.739889 15226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:05.740067 15226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:05.750106 15226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:05.750321 15226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:05.750579 15226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:03:05.766671 15226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.70.20:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:43157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=44269
--tserver_master_addrs=127.2.70.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:05.768714 15226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:05.770524 15226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:05.806010 15261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:05.806924 15262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:05.812583 15226 server_base.cc:1034] running on GCE node
W20260620 15:03:05.813285 15264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:05.814194 15226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:05.822919 15226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:05.824208 15226 hybrid_clock.cc:648] HybridClock initialized: now 1781967785824138 us; error 74 us; skew 500 ppm
I20260620 15:03:05.824585 15226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:05 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967785, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:05.828104 15226 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:03:05.830817 15226 webserver.cc:460] Webserver started at http://127.2.70.1:44269/ using document root <none> and password file <none>
I20260620 15:03:05.831559 15226 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:05.831722 15226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:03:05 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967785, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:05.862267 15226 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:03:05.874838 15272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:05.879524 15226 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.010s
I20260620 15:03:05.879797 15226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "e5c8164242dc4de1b9b76c727de464de"
format_stamp: "Formatted at 2026-06-20 15:03:02 on dist-test-slave-n5vv"
server_key: "602318fd103fc57b304ccc5141f1860d"
server_key_iv: "5bebfec13408031cd9ecd01dcf60556c"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:05.881385 15226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:05.928457 15226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:05.933777 15226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:05.934120 15226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:05.941421 15226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:05.949369 15226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:05.949595 15226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260620 15:03:05.949781 15226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:05.949906 15226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:06.025285 15226 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:43157
I20260620 15:03:06.025329 15385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:43157 every 8 connection(s)
I20260620 15:03:06.027107 15226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:03:06.036372 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 15226
I20260620 15:03:06.037521 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:38243
--local_ip_for_outbound_sockets=127.2.70.2
--tserver_master_addrs=127.2.70.62:45193
--webserver_port=44353
--webserver_interface=127.2.70.2
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.70.20:49563/kms with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:03:06 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967785, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:06.068166 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:06.031524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45193 (local address 127.2.70.1:42259)
0620 15:03:06.035714 (+ 4190us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:06.035766 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:06.038224 (+ 2458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:06.044186 (+ 5962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:06.044202 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:06.044931 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:06.045835 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:06.045867 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:06.048524 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:06.048531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:06.050182 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:06.050200 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:06.052326 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:06.052336 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:06.052646 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:06.053508 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:06.053548 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:06.056366 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:06.061819 (+ 5453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:06.061838 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:06.061861 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:06.062622 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:06.063238 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:06.063243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:06.063247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:06.063545 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:06.064896 (+ 1351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:06.064908 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:06.065435 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 15:03:06.065813 (+ 378us) client_negotiation.cc:240] Negotiation successful
0620 15:03:06.066508 (+ 695us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3874,"thread_start_us":3621,"threads_started":1}
I20260620 15:03:06.071015 15388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:06.031364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:42259 (local address 127.2.70.62:45193)
0620 15:03:06.042139 (+ 10775us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:06.042162 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:06.042225 (+ 63us) server_negotiation.cc:407] Connection header received
0620 15:03:06.042773 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:06.042799 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:06.043388 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:06.043933 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:06.046120 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:06.048297 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:06.050484 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:06.051648 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:06.051857 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:06.056757 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:06.056769 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:06.056793 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:06.056844 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:06.061594 (+ 4750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:06.062830 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:06.062839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:06.062848 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:06.063074 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:06.063713 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:06.063721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:06.063727 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:06.064567 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:06.064806 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:06.069131 (+ 4325us) server_negotiation.cc:299] Negotiation successful
0620 15:03:06.069579 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10392,"thread_start_us":10092,"threads_started":1}
I20260620 15:03:06.074970 15386 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45193
I20260620 15:03:06.075273 15386 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:06.076162 15386 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
W20260620 15:03:06.079372 15386 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:45193 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:03:06.079578 15386 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:06.079816 15386 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
I20260620 15:03:06.081080 15386 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:06.081354 15386 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
W20260620 15:03:06.082546 15386 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:03:06.298027 15397 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260620 15:03:06.480438 15390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:06.480751 15390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:06.480911 15390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:06.488479 15390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:06.488633 15390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:06.488801 15390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:03:06.499192 15390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.70.20:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:38243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=44353
--tserver_master_addrs=127.2.70.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:06.500859 15390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:06.502527 15390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:06.529093 15404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:06.536863 15405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:06.540014 15407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:06.540067 15390 server_base.cc:1034] running on GCE node
I20260620 15:03:06.541013 15390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:06.547685 15390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:06.548966 15390 hybrid_clock.cc:648] HybridClock initialized: now 1781967786548894 us; error 74 us; skew 500 ppm
I20260620 15:03:06.549345 15390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:06 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967786, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:06.552657 15390 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:03:06.554623 15390 webserver.cc:460] Webserver started at http://127.2.70.2:44353/ using document root <none> and password file <none>
I20260620 15:03:06.555354 15390 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:06.555528 15390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:03:06 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967786, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:06.588109 15390 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.007s sys 0.007s
I20260620 15:03:06.606225 15415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:06.608388 15390 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.005s sys 0.008s
I20260620 15:03:06.608716 15390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "fd8982c3eef941cca60f8a5e1fd47cd5"
format_stamp: "Formatted at 2026-06-20 15:03:03 on dist-test-slave-n5vv"
server_key: "34c868afc621b4a1b98f4c5999a9cacb"
server_key_iv: "fbe85815210d973c68b99cc85e988b40"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:06.610294 15390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:06.640038 15390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:06.644982 15390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:06.645282 15390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:06.646533 15390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:06.648021 15390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:06.648142 15390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:06.648283 15390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:06.648370 15390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:06.698652 15390 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:38243
I20260620 15:03:06.698689 15529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:38243 every 8 connection(s)
I20260620 15:03:06.700219 15390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:03:06.707623 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 15390
I20260620 15:03:06.708424 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.70.20:49563/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:44067
--local_ip_for_outbound_sockets=127.2.70.3
--tserver_master_addrs=127.2.70.62:45193
--webserver_port=45329
--webserver_interface=127.2.70.3
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:44715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.70.20:49563/kms with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:03:06 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781967786, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 15:03:06 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967786, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:06.734385 15533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:06.704043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45193 (local address 127.2.70.2:49493)
0620 15:03:06.708222 (+ 4179us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:06.708278 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:06.710791 (+ 2513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:06.711832 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:06.711847 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:06.712460 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:06.713344 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:06.713372 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:06.715021 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:06.715027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:06.717073 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:06.717089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:06.718410 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:06.718418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:06.718706 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:06.719538 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:06.719568 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:06.722508 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:06.728448 (+ 5940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:06.728459 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:06.728482 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:06.729255 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:06.729879 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:06.729886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:06.729892 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:06.730218 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:06.731097 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:06.731108 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:06.731635 (+ 527us) client_negotiation.cc:769] Sending connection context
0620 15:03:06.732007 (+ 372us) client_negotiation.cc:240] Negotiation successful
0620 15:03:06.732684 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3861,"thread_start_us":3627,"threads_started":1}
I20260620 15:03:06.735769 15532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:06.703590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:49493 (local address 127.2.70.62:45193)
0620 15:03:06.707811 (+ 4221us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:06.707819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:06.711227 (+ 3408us) server_negotiation.cc:407] Connection header received
0620 15:03:06.711302 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:06.711307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:06.711395 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:06.711522 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:06.713602 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:06.714818 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:06.717243 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:06.717945 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:06.718093 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:06.722788 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:06.722797 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:06.722805 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:06.722854 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:06.728226 (+ 5372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:06.729441 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:06.729450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:06.729456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:06.729669 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:06.730375 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:06.730383 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:06.730387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:06.730819 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:06.730993 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:06.735020 (+ 4027us) server_negotiation.cc:299] Negotiation successful
0620 15:03:06.735305 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4056,"thread_start_us":3933,"threads_started":1}
I20260620 15:03:06.738168 15530 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45193
I20260620 15:03:06.738482 15530 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:06.739359 15530 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
I20260620 15:03:06.741609 15189 ts_manager.cc:194] Registered new tserver with Master: fd8982c3eef941cca60f8a5e1fd47cd5 (127.2.70.2:38243)
I20260620 15:03:06.743798 15189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:49493
I20260620 15:03:07.082717 15386 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:07.083168 15386 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
I20260620 15:03:07.084612 15189 ts_manager.cc:194] Registered new tserver with Master: e5c8164242dc4de1b9b76c727de464de (127.2.70.1:43157)
I20260620 15:03:07.085855 15189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:42259
W20260620 15:03:07.113346 15534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:07.113755 15534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:07.113955 15534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:07.125232 15534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:07.125366 15534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:07.125631 15534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:03:07.140993 15534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:44715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.70.20:49563/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:44067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=45329
--tserver_master_addrs=127.2.70.62:45193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:07.142671 15534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:07.144363 15534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Jun 20 15:03:07 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967786, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
W20260620 15:03:07.166244 15541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:07.169818 15540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:07.172489 15543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:07.173259 15534 server_base.cc:1034] running on GCE node
I20260620 15:03:07.174144 15534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:07.175459 15534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:07.178915 15534 hybrid_clock.cc:648] HybridClock initialized: now 1781967787178714 us; error 184 us; skew 500 ppm
I20260620 15:03:07.179361 15534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:07 dist-test-slave-n5vv krb5kdc[14178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967787, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:07.185907 15534 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:03:07.192126 15534 webserver.cc:460] Webserver started at http://127.2.70.3:45329/ using document root <none> and password file <none>
I20260620 15:03:07.193679 15534 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:07.193935 15534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 15:03:07 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967787, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:07.278762 15534 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.005s sys 0.000s
I20260620 15:03:07.299079 15551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:07.303468 15534 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.010s sys 0.002s
I20260620 15:03:07.303766 15534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "246f025a3d2d45d592d720ce32897a42"
format_stamp: "Formatted at 2026-06-20 15:03:04 on dist-test-slave-n5vv"
server_key: "5a52cbb6163e097972ec4516dea363a4"
server_key_iv: "00a4a107f7039729652d21e080e370f4"
server_key_version: "kuduclusterkey@0"
I20260620 15:03:07.305389 15534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:07.400260 15534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:07.405620 15534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:07.405970 15534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:07.413357 15534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:07.421808 15534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:07.422019 15534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:07.422175 15534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:07.422283 15534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 15:03:07 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967786, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for HTTP/127.2.70.20@KRBTEST.COM
I20260620 15:03:07.716296 15534 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:44067
I20260620 15:03:07.716383 15664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:44067 every 8 connection(s)
I20260620 15:03:07.718369 15534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:03:07.723192 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 15534
I20260620 15:03:07.748816 15530 heartbeater.cc:503] Master 127.2.70.62:45193 was elected leader, sending a full tablet report...
Jun 20 15:03:07 dist-test-slave-n5vv krb5kdc[14178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967787, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:07.792476 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.748214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:45193 (local address 127.2.70.3:55643)
0620 15:03:07.758143 (+ 9929us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:07.758204 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:07.761178 (+ 2974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:07.767707 (+ 6529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:07.767723 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:07.768404 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:07.770728 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.770759 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.772355 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.772362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:07.774537 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.774554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.776929 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.776937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:07.777244 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.778113 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:07.778143 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:07.781012 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:07.787248 (+ 6236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:07.787259 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:07.787282 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:07.787968 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:07.788540 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:07.788545 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:07.788549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:07.788840 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:07.789633 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:07.789642 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:07.790116 (+ 474us) client_negotiation.cc:769] Sending connection context
0620 15:03:07.790427 (+ 311us) client_negotiation.cc:240] Negotiation successful
0620 15:03:07.790999 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9571,"thread_start_us":9498,"threads_started":1}
I20260620 15:03:07.795358 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.748894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:55643 (local address 127.2.70.62:45193)
0620 15:03:07.767014 (+ 18120us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:07.767021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:07.767043 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:03:07.767101 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:07.767106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:07.767184 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:07.767318 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:07.770985 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.772156 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.774751 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.776484 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.776606 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.781353 (+ 4747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:07.781360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:07.781383 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:07.781431 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:07.786987 (+ 5556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:07.788148 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:07.788153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:07.788157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:07.788378 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:07.788993 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:07.788998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:07.789001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:07.789374 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:07.789557 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:07.792997 (+ 3440us) server_negotiation.cc:299] Negotiation successful
0620 15:03:07.794943 (+ 1946us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17960,"thread_start_us":16589,"threads_started":1}
I20260620 15:03:07.795977 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.724320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.2.70.62:45193)
0620 15:03:07.762016 (+ 37696us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:07.762022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:07.762044 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:03:07.762111 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:07.762116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:07.762194 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:07.762305 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:07.768812 (+ 6507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.769936 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.774921 (+ 4985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.775600 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.775729 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.781795 (+ 6066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:07.781802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:07.781807 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:07.781885 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:07.786554 (+ 4669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:07.793074 (+ 6520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:07.793079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:07.793082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:07.793236 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:07.794005 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:07.794010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:07.794013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:07.794402 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:07.794708 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:07.795066 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 15:03:07.795276 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37559,"thread_start_us":42386,"threads_started":1}
I20260620 15:03:07.797186 15665 heartbeater.cc:347] Connected to a master server at 127.2.70.62:45193
I20260620 15:03:07.797593 15665 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:07.798537 15665 heartbeater.cc:511] Master 127.2.70.62:45193 requested a full tablet report, sending...
I20260620 15:03:07.800182 15189 ts_manager.cc:194] Registered new tserver with Master: 246f025a3d2d45d592d720ce32897a42 (127.2.70.3:44067)
I20260620 15:03:07.801301 15189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:55643
I20260620 15:03:07.812147 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:03:07.834131 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.819859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.2.70.62:45193)
0620 15:03:07.820125 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:07.820134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:07.820685 (+ 551us) server_negotiation.cc:407] Connection header received
0620 15:03:07.820929 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:07.820937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:07.821011 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:07.821123 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:07.822708 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.823562 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.824941 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.825388 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.825567 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.828300 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:07.828312 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:07.828315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:07.828351 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:07.831094 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:07.831960 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:07.831966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:07.831974 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:07.832108 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:07.832493 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:07.832499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:07.832506 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:07.832822 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:07.833073 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:07.833542 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 15:03:07.833775 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:03:07.839874 15189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:03:07.848981 15189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:03:07.893754 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.873283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:38243 (local address 127.0.0.1:48118)
0620 15:03:07.881585 (+ 8302us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:07.881606 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:07.884035 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:07.885001 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:07.885005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:07.885029 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:07.885429 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.885452 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.887573 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.887578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:07.890427 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.890453 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.892726 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.892733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:07.892951 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.893011 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 15:03:07.893137 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 15:03:07.893342 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8153,"thread_start_us":4958,"threads_started":1}
I20260620 15:03:07.894820 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.874007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48118 (local address 127.2.70.2:38243)
0620 15:03:07.878026 (+ 4019us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:07.878037 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:07.881697 (+ 3660us) server_negotiation.cc:407] Connection header received
0620 15:03:07.884246 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:07.884256 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:07.884539 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:07.884713 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:07.885658 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.887349 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.890620 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.892143 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.892244 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.892505 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:07.894300 (+ 1795us) server_negotiation.cc:299] Negotiation successful
0620 15:03:07.894497 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3821,"thread_start_us":3653,"threads_started":1}
I20260620 15:03:07.900350 15465 tablet_service.cc:1467] Processing CreateTablet for tablet 4725199b15ac47f0af0b55d7a9004723 (DEFAULT_TABLE table=test-table [id=b8d84b3a61ee4730aa06d37486f6128e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:07.901959 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.871836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:44067 (local address 127.0.0.1:46272)
0620 15:03:07.878662 (+ 6826us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:07.878706 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:07.878915 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:07.880524 (+ 1609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:07.880532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:07.880565 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:07.880969 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.880982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.896969 (+ 15987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.896975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:07.899133 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.899149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.901134 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.901141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:07.901308 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.901343 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:03:07.901434 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 15:03:07.901603 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6535,"thread_start_us":9435,"threads_started":1}
I20260620 15:03:07.903021 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.872725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46272 (local address 127.2.70.3:44067)
0620 15:03:07.879484 (+ 6759us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:07.879492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:07.879522 (+ 30us) server_negotiation.cc:407] Connection header received
0620 15:03:07.879628 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:07.879636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:07.879892 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:07.880099 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:07.881212 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.896790 (+ 15578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.899359 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.900884 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.902219 (+ 1335us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.902467 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:07.902539 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 15:03:07.902697 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6577,"thread_start_us":7741,"threads_started":1}
I20260620 15:03:07.905196 15465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4725199b15ac47f0af0b55d7a9004723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:07.913228 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.881964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:57964)
0620 15:03:07.894028 (+ 12064us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:07.894048 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:07.894140 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:07.905644 (+ 11504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:07.905649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:07.905679 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:07.905845 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.905854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.908025 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.908030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:07.910107 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.910122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.912467 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.912474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:07.912667 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.912705 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:03:07.912776 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 15:03:07.912937 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11944,"thread_start_us":30169,"threads_started":1}
I20260620 15:03:07.914296 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.882044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.2.70.1:43157)
0620 15:03:07.903683 (+ 21639us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:07.903692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:07.903730 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:03:07.903830 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:07.903838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:07.904150 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:07.906029 (+ 1879us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:07.906289 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.907861 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.910396 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.911909 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.913470 (+ 1561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:07.913733 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:07.913809 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 15:03:07.913967 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21453,"thread_start_us":30229,"threads_started":1}
I20260620 15:03:07.916594 15600 tablet_service.cc:1467] Processing CreateTablet for tablet 4725199b15ac47f0af0b55d7a9004723 (DEFAULT_TABLE table=test-table [id=b8d84b3a61ee4730aa06d37486f6128e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:07.918195 15600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4725199b15ac47f0af0b55d7a9004723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:07.918154 15321 tablet_service.cc:1467] Processing CreateTablet for tablet 4725199b15ac47f0af0b55d7a9004723 (DEFAULT_TABLE table=test-table [id=b8d84b3a61ee4730aa06d37486f6128e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:07.919888 15321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4725199b15ac47f0af0b55d7a9004723. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:07.939651 15693 tablet_bootstrap.cc:492] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: Bootstrap starting.
I20260620 15:03:07.941627 15695 tablet_bootstrap.cc:492] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de: Bootstrap starting.
I20260620 15:03:07.943854 15694 tablet_bootstrap.cc:492] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: Bootstrap starting.
I20260620 15:03:07.946332 15695 tablet_bootstrap.cc:654] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:07.947880 15695 log.cc:824] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:07.948143 15694 tablet_bootstrap.cc:654] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:07.948179 15693 tablet_bootstrap.cc:654] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:07.949443 15693 log.cc:824] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:07.949833 15694 log.cc:824] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:07.960904 15693 tablet_bootstrap.cc:492] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: No bootstrap required, opened a new log
I20260620 15:03:07.961243 15693 ts_tablet_manager.cc:1397] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.004s
I20260620 15:03:07.961732 15695 tablet_bootstrap.cc:492] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de: No bootstrap required, opened a new log
I20260620 15:03:07.961766 15694 tablet_bootstrap.cc:492] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: No bootstrap required, opened a new log
I20260620 15:03:07.962040 15695 ts_tablet_manager.cc:1397] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.004s
I20260620 15:03:07.962244 15694 ts_tablet_manager.cc:1397] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.011s
I20260620 15:03:07.971995 15693 raft_consensus.cc:348] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:07.972498 15693 raft_consensus.cc:374] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:07.972604 15693 raft_consensus.cc:729] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8982c3eef941cca60f8a5e1fd47cd5, State: Initialized, Role: FOLLOWER
I20260620 15:03:07.973402 15693 consensus_queue.cc:260] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:07.974287 15694 raft_consensus.cc:348] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:07.974793 15694 raft_consensus.cc:374] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:07.974951 15694 raft_consensus.cc:729] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 246f025a3d2d45d592d720ce32897a42, State: Initialized, Role: FOLLOWER
I20260620 15:03:07.975728 15694 consensus_queue.cc:260] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:07.976015 15695 raft_consensus.cc:348] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:07.976526 15695 raft_consensus.cc:374] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:07.976661 15695 raft_consensus.cc:729] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c8164242dc4de1b9b76c727de464de, State: Initialized, Role: FOLLOWER
I20260620 15:03:07.977516 15695 consensus_queue.cc:260] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:07.988548 15695 ts_tablet_manager.cc:1428] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de: Time spent starting tablet: real 0.026s user 0.010s sys 0.012s
I20260620 15:03:07.989022 15386 heartbeater.cc:503] Master 127.2.70.62:45193 was elected leader, sending a full tablet report...
I20260620 15:03:07.991604 15701 raft_consensus.cc:482] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:03:07.992029 15701 raft_consensus.cc:504] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:07.993727 15665 heartbeater.cc:503] Master 127.2.70.62:45193 was elected leader, sending a full tablet report...
I20260620 15:03:07.995594 15693 ts_tablet_manager.cc:1428] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: Time spent starting tablet: real 0.034s user 0.020s sys 0.016s
I20260620 15:03:08.002027 15694 ts_tablet_manager.cc:1428] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: Time spent starting tablet: real 0.040s user 0.010s sys 0.008s
I20260620 15:03:08.003480 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.995390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:38243 (local address 127.2.70.3:49433)
0620 15:03:07.995851 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:07.995875 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:07.996015 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:07.997037 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:07.997042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:07.997070 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:07.997261 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:07.997273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.998885 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:07.998892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:08.000873 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:08.000889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.002676 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.002682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:08.002897 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:08.002930 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:03:08.003038 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:03:08.003179 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327}
I20260620 15:03:08.003512 15701 leader_election.cc:288] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd8982c3eef941cca60f8a5e1fd47cd5 (127.2.70.2:38243), e5c8164242dc4de1b9b76c727de464de (127.2.70.1:43157)
I20260620 15:03:08.004057 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:07.996338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:49433 (local address 127.2.70.2:38243)
0620 15:03:07.996566 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:07.996572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:07.996594 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:03:07.996667 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:07.996674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:07.996770 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:07.996868 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:07.997459 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:07.998723 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.001083 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.002368 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.002490 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:08.002589 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:08.003678 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0620 15:03:08.003816 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:03:08.005084 15485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4725199b15ac47f0af0b55d7a9004723" candidate_uuid: "246f025a3d2d45d592d720ce32897a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" is_pre_election: true
I20260620 15:03:08.005666 15485 raft_consensus.cc:2454] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 246f025a3d2d45d592d720ce32897a42 in term 0.
I20260620 15:03:08.006520 15553 leader_election.cc:302] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 246f025a3d2d45d592d720ce32897a42, fd8982c3eef941cca60f8a5e1fd47cd5; no voters:
I20260620 15:03:08.007055 15701 raft_consensus.cc:2781] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:03:08.007218 15701 raft_consensus.cc:482] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:03:08.007320 15701 raft_consensus.cc:3037] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:08.010211 15701 raft_consensus.cc:504] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:08.010815 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.000097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:54159 (local address 127.2.70.1:43157)
0620 15:03:08.000286 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:08.000293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:08.002225 (+ 1932us) server_negotiation.cc:407] Connection header received
0620 15:03:08.002510 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:08.002516 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:08.002609 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:08.002678 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:08.004711 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.005666 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.008428 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.009665 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.009799 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:08.009898 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:08.010396 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 15:03:08.010561 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:03:08.011354 15701 leader_election.cc:288] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [CANDIDATE]: Term 1 election: Requested vote from peers fd8982c3eef941cca60f8a5e1fd47cd5 (127.2.70.2:38243), e5c8164242dc4de1b9b76c727de464de (127.2.70.1:43157)
I20260620 15:03:08.011461 15485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4725199b15ac47f0af0b55d7a9004723" candidate_uuid: "246f025a3d2d45d592d720ce32897a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5"
I20260620 15:03:08.011667 15485 raft_consensus.cc:3037] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:08.012682 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.000293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.2.70.3:54159)
0620 15:03:08.002117 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.002144 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:08.002328 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:08.002885 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:08.002892 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:08.002921 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:08.004512 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:08.004528 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.005865 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.005873 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:08.008252 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:08.008267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.009886 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.009896 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:08.010135 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:08.010185 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 15:03:08.011938 (+ 1753us) client_negotiation.cc:240] Negotiation successful
0620 15:03:08.012192 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1672,"thread_start_us":1859,"threads_started":1}
I20260620 15:03:08.013907 15341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4725199b15ac47f0af0b55d7a9004723" candidate_uuid: "246f025a3d2d45d592d720ce32897a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c8164242dc4de1b9b76c727de464de" is_pre_election: true
I20260620 15:03:08.014357 15341 raft_consensus.cc:2454] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 246f025a3d2d45d592d720ce32897a42 in term 0.
I20260620 15:03:08.014765 15485 raft_consensus.cc:2454] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 246f025a3d2d45d592d720ce32897a42 in term 1.
I20260620 15:03:08.014916 15340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4725199b15ac47f0af0b55d7a9004723" candidate_uuid: "246f025a3d2d45d592d720ce32897a42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c8164242dc4de1b9b76c727de464de"
I20260620 15:03:08.015161 15553 leader_election.cc:302] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 246f025a3d2d45d592d720ce32897a42, fd8982c3eef941cca60f8a5e1fd47cd5; no voters:
I20260620 15:03:08.015172 15340 raft_consensus.cc:3037] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:08.015607 15701 raft_consensus.cc:2781] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:08.015832 15701 raft_consensus.cc:686] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 1 LEADER]: Becoming Leader. State: Replica: 246f025a3d2d45d592d720ce32897a42, State: Running, Role: LEADER
I20260620 15:03:08.016462 15701 consensus_queue.cc:237] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } }
I20260620 15:03:08.018275 15340 raft_consensus.cc:2454] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 246f025a3d2d45d592d720ce32897a42 in term 1.
I20260620 15:03:08.022162 15188 catalog_manager.cc:5495] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 246f025a3d2d45d592d720ce32897a42 (127.2.70.3). New cstate: current_term: 1 leader_uuid: "246f025a3d2d45d592d720ce32897a42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "246f025a3d2d45d592d720ce32897a42" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 44067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 } health_report { overall_health: UNKNOWN } } }
W20260620 15:03:08.036305 15387 tablet.cc:2369] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:03:08.055509 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.050263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46280 (local address 127.2.70.3:44067)
0620 15:03:08.050481 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:08.050488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:08.050508 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:03:08.050619 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:08.050623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:08.050699 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:08.050853 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:03:08.051249 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.051973 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.053323 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.053796 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.053924 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:08.054424 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:03:08.054924 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:03:08.055107 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:08.055157 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 15:03:08.055238 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:03:08.063799 15485 raft_consensus.cc:1261] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 1 FOLLOWER]: Refusing update from remote peer 246f025a3d2d45d592d720ce32897a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:08.064023 15340 raft_consensus.cc:1261] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de [term 1 FOLLOWER]: Refusing update from remote peer 246f025a3d2d45d592d720ce32897a42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:08.064785 15701 consensus_queue.cc:1035] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd8982c3eef941cca60f8a5e1fd47cd5" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 38243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:08.065074 15700 consensus_queue.cc:1035] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c8164242dc4de1b9b76c727de464de" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 43157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:08.076710 15706 mvcc.cc:205] Tried to move back new op lower bound from 7298940059898294272 to 7298940059720384512. Current Snapshot: MvccSnapshot[applied={T|T < 7298940059898294272}]
I20260620 15:03:08.078898 15707 mvcc.cc:205] Tried to move back new op lower bound from 7298940059898294272 to 7298940059720384512. Current Snapshot: MvccSnapshot[applied={T|T < 7298940059898294272}]
I20260620 15:03:08.103125 15189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:03:08.103443 15189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:03:08.107488 15189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 243b0a74afdc45ab8aa87c629dc75eb5: Sending DeleteTablet for 3 replicas of tablet 4725199b15ac47f0af0b55d7a9004723
I20260620 15:03:08.108657 15465 tablet_service.cc:1514] Processing DeleteTablet for tablet 4725199b15ac47f0af0b55d7a9004723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:48118
I20260620 15:03:08.108873 15600 tablet_service.cc:1514] Processing DeleteTablet for tablet 4725199b15ac47f0af0b55d7a9004723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:46272
I20260620 15:03:08.109537 15321 tablet_service.cc:1514] Processing DeleteTablet for tablet 4725199b15ac47f0af0b55d7a9004723 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:08 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:57964
I20260620 15:03:08.114202 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 15226
I20260620 15:03:08.138541 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.116673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57978 (local address 127.2.70.1:43157)
0620 15:03:08.116894 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:08.116904 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:08.116930 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:03:08.117020 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:08.117029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:08.117149 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:08.117285 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:08.118946 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.120452 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.123250 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.124217 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.124372 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:08.127644 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:08.127666 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:08.127693 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:08.127747 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:08.132754 (+ 5007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:08.134155 (+ 1401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:08.134169 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:08.134176 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:08.134382 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:08.134925 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:08.134933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:08.134939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:08.135519 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:08.135909 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:08.137593 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0620 15:03:08.137895 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:03:08.139302 15381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:08.158070 15717 tablet_replica.cc:330] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: stopping tablet replica
I20260620 15:03:08.158597 15717 raft_consensus.cc:2229] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:03:08.159246 15717 raft_consensus.cc:2258] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:08.167393 15717 ts_tablet_manager.cc:1905] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:08.174050 15716 tablet_replica.cc:330] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: stopping tablet replica
I20260620 15:03:08.174531 15716 raft_consensus.cc:2229] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:03:08.175074 15716 raft_consensus.cc:2258] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:08.178390 15716 ts_tablet_manager.cc:1905] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:08.183799 15717 ts_tablet_manager.cc:1918] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:08.183976 15717 log.cc:1196] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/4725199b15ac47f0af0b55d7a9004723
I20260620 15:03:08.184495 15717 ts_tablet_manager.cc:1939] T 4725199b15ac47f0af0b55d7a9004723 P 246f025a3d2d45d592d720ce32897a42: Deleting consensus metadata
I20260620 15:03:08.185838 15716 ts_tablet_manager.cc:1918] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:08.186084 15716 log.cc:1196] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/4725199b15ac47f0af0b55d7a9004723
I20260620 15:03:08.186365 15176 catalog_manager.cc:4841] TS 246f025a3d2d45d592d720ce32897a42 (127.2.70.3:44067): tablet 4725199b15ac47f0af0b55d7a9004723 (table test-table [id=b8d84b3a61ee4730aa06d37486f6128e]) successfully deleted
I20260620 15:03:08.186689 15716 ts_tablet_manager.cc:1939] T 4725199b15ac47f0af0b55d7a9004723 P fd8982c3eef941cca60f8a5e1fd47cd5: Deleting consensus metadata
I20260620 15:03:08.188344 15174 catalog_manager.cc:4841] TS fd8982c3eef941cca60f8a5e1fd47cd5 (127.2.70.2:38243): tablet 4725199b15ac47f0af0b55d7a9004723 (table test-table [id=b8d84b3a61ee4730aa06d37486f6128e]) successfully deleted
I20260620 15:03:08.260114 15718 tablet_replica.cc:330] T 4725199b15ac47f0af0b55d7a9004723 P e5c8164242dc4de1b9b76c727de464de: stopping tablet replica
I20260620 15:03:08.260145 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 15226
W20260620 15:03:08.282655 15176 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.1:43157 (error 108)
I20260620 15:03:08.289203 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 15390
I20260620 15:03:08.292085 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.291571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:57994)
0620 15:03:08.291737 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.291866 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:03:08.292469 15176 catalog_manager.cc:4584] TS e5c8164242dc4de1b9b76c727de464de (127.2.70.1:43157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4725199b15ac47f0af0b55d7a9004723: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
I20260620 15:03:08.312688 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.290486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.2.70.2:38243)
0620 15:03:08.290679 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:08.290685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:08.290707 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:03:08.290772 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:08.290778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:08.290853 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:08.290977 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:08.292784 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.293935 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.295868 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.296472 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.296592 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:08.305341 (+ 8749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:08.305363 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:08.305386 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:08.305450 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:08.309379 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:08.310391 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:08.310401 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:08.310404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:08.310567 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:08.310988 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:08.310993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:08.310997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:08.311450 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:08.311783 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:08.312052 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 15:03:08.312285 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:03:08.313441 15525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:08.353184 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.352634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:57996)
0620 15:03:08.352890 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.353010 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 15:03:08.354458 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.354066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:57998)
0620 15:03:08.354227 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.354315 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 15:03:08.415163 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.414596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:58000)
0620 15:03:08.414852 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.414981 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 15:03:08.416494 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.416050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:58012)
0620 15:03:08.416231 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.416322 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:03:08.419291 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 15390
I20260620 15:03:08.437345 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 15534
I20260620 15:03:08.453699 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.438407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.2.70.3:44067)
0620 15:03:08.438614 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:08.438620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:08.438669 (+ 49us) server_negotiation.cc:407] Connection header received
0620 15:03:08.438885 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:08.438893 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:08.439028 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:08.439152 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:08.440561 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.441870 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.443362 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:08.444138 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:08.444266 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:08.446808 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:08.446823 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:08.446847 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:08.446898 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:08.450691 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:08.451616 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:08.451623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:08.451625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:08.451787 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:08.452199 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:08.452202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:08.452204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:08.452585 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:08.452825 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:08.453162 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 15:03:08.453337 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:03:08.454528 15660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:08.497377 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.496853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:58014)
0620 15:03:08.497116 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.497216 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 15:03:08.498777 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.498270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:58026)
0620 15:03:08.498520 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.498619 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260620 15:03:08.578701 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 15534
I20260620 15:03:08.597385 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 15156
I20260620 15:03:08.598654 15219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:08.711592 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:08.710880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:43157 (local address 127.0.0.1:58036)
0620 15:03:08.711183 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:08.711301 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260620 15:03:08.711889 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 15156
I20260620 15:03:08.733418 2328 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 15:03:09.069015 2328 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 15:03:09.069217 2328 mini_ranger.cc:66] Stopping Ranger...
I20260620 15:03:09.414942 2328 mini_ranger.cc:68] Stopped Ranger
2026-06-20 15:03:09.415 UTC [14208] LOG: received smart shutdown request
2026-06-20 15:03:09.418 UTC [14208] LOG: background worker "logical replication launcher" (PID 14217) exited with exit code 1
2026-06-20 15:03:09.418 UTC [14212] LOG: shutting down
2026-06-20 15:03:09.598 UTC [14208] LOG: database system is shut down
2026-06-20T15:03:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53514 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:03:09 dist-test-slave-n5vv krb5kdc[15732](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:03:09 dist-test-slave-n5vv krb5kdc[15732](info): set up 2 sockets
Jun 20 15:03:09 dist-test-slave-n5vv krb5kdc[15732](info): commencing operation
krb5kdc: starting...
W20260620 15:03:10.865873 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.010s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:03:11 dist-test-slave-n5vv krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:03:11.097326 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46349
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:46349,127.2.70.61:40533,127.2.70.60:39835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:11.428280 15745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:11.428637 15745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:11.428819 15745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:11.436012 15745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:11.436172 15745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:11.436257 15745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:11.436367 15745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:11.436453 15745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:11.446522 15745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:46349,127.2.70.61:40533,127.2.70.60:39835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:11.447939 15745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:11.449306 15745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:11.461661 15750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:11.461753 15751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:11.461831 15753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:11.462859 15745 server_base.cc:1034] running on GCE node
I20260620 15:03:11.463729 15745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:11.465238 15745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:11.466471 15745 hybrid_clock.cc:648] HybridClock initialized: now 1781967791466419 us; error 84 us; skew 500 ppm
I20260620 15:03:11.466847 15745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:11 dist-test-slave-n5vv krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:11.470065 15745 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:03:11.471568 15745 webserver.cc:460] Webserver started at http://127.2.70.62:45911/ using document root <none> and password file <none>
I20260620 15:03:11.472286 15745 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:11.472430 15745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:11.472739 15745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:11.475787 15745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "001065394053451196f47ddb2a2d3dc0"
format_stamp: "Formatted at 2026-06-20 15:03:11 on dist-test-slave-n5vv"
server_key: "554ec524c5a6d0dea8d5892ee2198d66"
server_key_iv: "b2c5178a3d38fb19af128c778fcff525"
server_key_version: "encryptionkey@0"
I20260620 15:03:11.476609 15745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "001065394053451196f47ddb2a2d3dc0"
format_stamp: "Formatted at 2026-06-20 15:03:11 on dist-test-slave-n5vv"
server_key: "554ec524c5a6d0dea8d5892ee2198d66"
server_key_iv: "b2c5178a3d38fb19af128c778fcff525"
server_key_version: "encryptionkey@0"
I20260620 15:03:11.481859 15745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:03:11.485170 15760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:11.486562 15745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:11.486759 15745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "001065394053451196f47ddb2a2d3dc0"
format_stamp: "Formatted at 2026-06-20 15:03:11 on dist-test-slave-n5vv"
server_key: "554ec524c5a6d0dea8d5892ee2198d66"
server_key_iv: "b2c5178a3d38fb19af128c778fcff525"
server_key_version: "encryptionkey@0"
I20260620 15:03:11.486968 15745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:11.531132 15745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:11.535432 15745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:11.535768 15745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:11.554791 15745 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:46349
I20260620 15:03:11.554804 15811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:46349 every 8 connection(s)
I20260620 15:03:11.556187 15745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:03:11.560518 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 15745
I20260620 15:03:11.560755 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:03:11.560861 15812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:11.561547 2328 external_mini_cluster.cc:1428] Setting key 7f64ef0eef8cfaf482ffa304c833a74c
I20260620 15:03:11.567632 15812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.70.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.61@KRBTEST.COM" created.
Jun 20 15:03:11 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:11.596269 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.570957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.2.70.62:46349)
0620 15:03:11.576720 (+ 5763us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:11.576725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:11.576757 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:03:11.576895 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:11.576901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:11.577152 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:11.577314 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:11.579203 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:11.580460 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:11.582287 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:11.583079 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:11.583275 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:11.587125 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:11.587139 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:11.587161 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:11.587208 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:11.591268 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:11.592301 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:11.592309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:11.592311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:11.592666 (+ 355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:11.593198 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:11.593202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:11.593204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:11.593842 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:11.594019 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:11.594800 (+ 781us) server_negotiation.cc:299] Negotiation successful
0620 15:03:11.595106 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5579,"spinlock_wait_cycles":29568,"thread_start_us":5347,"threads_started":1}
I20260620 15:03:11.596328 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.570958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46349 (local address 127.0.0.1:36290)
0620 15:03:11.571993 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.572065 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:11.575695 (+ 3630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:11.577442 (+ 1747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:11.577452 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:11.578390 (+ 938us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:11.578968 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:11.578996 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:11.580640 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:11.580648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:11.582119 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:11.582131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:11.583369 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:11.583381 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:11.583663 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:11.584390 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:11.584413 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:11.586851 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:11.591408 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:11.591424 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:11.591440 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:11.592109 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:11.592802 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:11.592809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:11.592816 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:11.593028 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:11.594001 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:11.594013 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:11.594329 (+ 316us) client_negotiation.cc:769] Sending connection context
0620 15:03:11.594618 (+ 289us) client_negotiation.cc:240] Negotiation successful
0620 15:03:11.595318 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":120,"thread_start_us":414,"threads_started":1}
I20260620 15:03:11.599829 15812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:03:11.601106 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.600539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45480)
0620 15:03:11.600753 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.600901 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260620 15:03:11.601473 15764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
I20260620 15:03:11.603407 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.602891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45486)
0620 15:03:11.603116 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.603219 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 15:03:11.603706 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:40533: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab.
Entry for principal kudu/127.2.70.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab.
I20260620 15:03:11.616413 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.61:40533
--webserver_interface=127.2.70.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab
--principal=kudu/127.2.70.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:46349,127.2.70.61:40533,127.2.70.60:39835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:03:11.653083 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } attempt: 1
I20260620 15:03:11.654461 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.653844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45490)
0620 15:03:11.654129 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.654239 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260620 15:03:11.655653 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.655222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45506)
0620 15:03:11.655393 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.655490 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:03:11.655928 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:40533: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
I20260620 15:03:11.724220 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } attempt: 2
I20260620 15:03:11.725669 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.725027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45510)
0620 15:03:11.725325 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.725438 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260620 15:03:11.726873 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.726443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45526)
0620 15:03:11.726618 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.726718 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:03:11.727087 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:40533: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
I20260620 15:03:11.818352 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } attempt: 3
I20260620 15:03:11.819677 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.819069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45532)
0620 15:03:11.819332 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.819466 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 15:03:11.820714 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.820327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45536)
0620 15:03:11.820473 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.820567 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 15:03:11.820957 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:40533: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
W20260620 15:03:11.945938 15819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:11.946256 15819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:11.946396 15819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:11.954100 15819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:11.954212 15819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:11.954262 15819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:11.954319 15819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:11.954361 15819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:11.964229 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } attempt: 4
I20260620 15:03:11.964385 15819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:46349,127.2.70.61:40533,127.2.70.60:39835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab
--principal=kudu/127.2.70.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.61:40533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.70.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:11.965562 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.964919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45542)
0620 15:03:11.965251 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.965340 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260620 15:03:11.966082 15819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:11.966714 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:11.966343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45546)
0620 15:03:11.966491 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:11.966591 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:03:11.966908 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:40533: Network error: Client connection negotiation failed: client connection to 127.2.70.61:40533: connect: Connection refused (error 111)
I20260620 15:03:11.967628 15819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:11.979460 15824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:11.979563 15827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:11.979585 15825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:11.980237 15819 server_base.cc:1034] running on GCE node
I20260620 15:03:11.981086 15819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:11.982555 15819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:11.983754 15819 hybrid_clock.cc:648] HybridClock initialized: now 1781967791983708 us; error 44 us; skew 500 ppm
I20260620 15:03:11.984062 15819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:11 dist-test-slave-n5vv krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:11.987340 15819 init.cc:377] Logged in from keytab as kudu/127.2.70.61@KRBTEST.COM (short username kudu)
I20260620 15:03:11.988677 15819 webserver.cc:460] Webserver started at http://127.2.70.61:39053/ using document root <none> and password file <none>
I20260620 15:03:11.989629 15819 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:11.989841 15819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:11.990156 15819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:11.993173 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/data/instance:
uuid: "83513505386d4940a523a32a76fd22a5"
format_stamp: "Formatted at 2026-06-20 15:03:11 on dist-test-slave-n5vv"
server_key: "8b0055e39e3134efc2f7d7b09375f313"
server_key_iv: "05576ffcaf2d6e5ce8df3b997f4ebbe1"
server_key_version: "encryptionkey@0"
I20260620 15:03:11.994024 15819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/wal/instance:
uuid: "83513505386d4940a523a32a76fd22a5"
format_stamp: "Formatted at 2026-06-20 15:03:11 on dist-test-slave-n5vv"
server_key: "8b0055e39e3134efc2f7d7b09375f313"
server_key_iv: "05576ffcaf2d6e5ce8df3b997f4ebbe1"
server_key_version: "encryptionkey@0"
I20260620 15:03:11.999254 15819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:03:12.002696 15834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:12.004041 15819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:03:12.004240 15819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/wal
uuid: "83513505386d4940a523a32a76fd22a5"
format_stamp: "Formatted at 2026-06-20 15:03:11 on dist-test-slave-n5vv"
server_key: "8b0055e39e3134efc2f7d7b09375f313"
server_key_iv: "05576ffcaf2d6e5ce8df3b997f4ebbe1"
server_key_version: "encryptionkey@0"
I20260620 15:03:12.004451 15819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:12.035246 15819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:12.039579 15819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:12.039819 15819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:12.057675 15819 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.61:40533
I20260620 15:03:12.057698 15885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.61:40533 every 8 connection(s)
I20260620 15:03:12.059010 15819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
I20260620 15:03:12.062719 15886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:12.068653 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 15819
I20260620 15:03:12.068853 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-1/wal/instance
I20260620 15:03:12.069271 2328 external_mini_cluster.cc:1428] Setting key a12a7fc9b41b1ec5e8ddfd9ab95fd939
I20260620 15:03:12.069178 15886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.60@KRBTEST.COM; defaulting to no policy
I20260620 15:03:12.089460 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.071777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.2.70.62:46349)
0620 15:03:12.071995 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.072002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.073304 (+ 1302us) server_negotiation.cc:407] Connection header received
0620 15:03:12.075241 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.075247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.075340 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.075453 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.076923 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.077675 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.079209 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.079627 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.079766 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.083358 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.083363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.083366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.083401 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.086304 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.087062 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.087067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.087069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.087191 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.087666 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.087671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.087675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.087957 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.088132 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.088900 (+ 768us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.089139 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.60@KRBTEST.COM" created.
I20260620 15:03:12.090746 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.072302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46349 (local address 127.0.0.1:36294)
0620 15:03:12.073111 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.073157 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:12.074969 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:12.075697 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:12.075705 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:12.076198 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:12.076752 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.076770 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.077838 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.077842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.079050 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.079062 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.079852 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.079860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.080157 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.080841 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:12.080864 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:12.083123 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:12.086458 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.086467 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.086484 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.086919 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.087302 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.087307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.087311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.087538 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.088092 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:12.088101 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:12.088531 (+ 430us) client_negotiation.cc:769] Sending connection context
0620 15:03:12.088811 (+ 280us) client_negotiation.cc:240] Negotiation successful
0620 15:03:12.089318 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":351,"threads_started":1}
I20260620 15:03:12.092623 15886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:03:12.110821 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.093663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.2.70.61:40533)
0620 15:03:12.094483 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.094493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.094526 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:03:12.094584 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.094595 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.094831 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.094986 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.095642 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.097396 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.099307 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.099841 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.099932 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.103654 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.103671 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.103693 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.103735 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.107727 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.108419 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.108430 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.108437 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.108602 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.109041 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.109045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.109048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.109434 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.109672 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.110107 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.110489 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":619,"thread_start_us":437,"threads_started":1}
I20260620 15:03:12.110854 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.093526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45562)
0620 15:03:12.093758 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.093792 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:12.093902 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:12.094986 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:12.094989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:12.095352 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:12.095484 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.095492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.097614 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.097620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.099151 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.099167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.099989 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.099998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.100264 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.100843 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:12.100869 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:12.103428 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:12.107893 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.107896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.107900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.108259 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.108725 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.108730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.108734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.108911 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.109615 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:12.109627 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:12.109827 (+ 200us) client_negotiation.cc:769] Sending connection context
0620 15:03:12.110011 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 15:03:12.110378 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 15:03:12.113214 15886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:03:12.114323 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.113855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:42972)
0620 15:03:12.113998 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.114111 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:03:12.114624 15838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
I20260620 15:03:12.116140 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.115851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:42986)
0620 15:03:12.115976 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.116047 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:03:12.116385 15886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab.
Entry for principal kudu/127.2.70.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab.
I20260620 15:03:12.121436 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.60:39835
--webserver_interface=127.2.70.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab
--principal=kudu/127.2.70.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:46349,127.2.70.61:40533,127.2.70.60:39835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:03:12.165680 15886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 1
I20260620 15:03:12.166891 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.166431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43000)
0620 15:03:12.166632 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.166745 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 15:03:12.167877 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.167526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43008)
0620 15:03:12.167690 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.167772 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 15:03:12.168097 15886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
I20260620 15:03:12.236339 15886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 2
I20260620 15:03:12.237811 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.237171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43018)
0620 15:03:12.237526 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.237644 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 15:03:12.238950 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.238571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43032)
0620 15:03:12.238741 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.238816 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 15:03:12.239182 15886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
I20260620 15:03:12.266170 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } attempt: 5
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:03:12.280529 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.266985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45578)
0620 15:03:12.267207 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.267230 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:12.267356 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:12.267719 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:12.267744 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:12.268131 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:12.268298 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.268306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.269756 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.269760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.270586 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.270596 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.271553 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.271557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.271701 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.272345 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:12.272364 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:12.274452 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:12.278354 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.278358 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.278360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.278724 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.279116 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.279119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.279122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.279246 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.279842 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:12.279845 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:12.279981 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 15:03:12.280090 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:03:12.280282 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 15:03:12.280956 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.267164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.2.70.61:40533)
0620 15:03:12.267367 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.267373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.267400 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:03:12.267470 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.267476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.267576 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.267691 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.268461 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.269589 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.270730 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.271410 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.271536 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.274656 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.274662 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.274667 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.274706 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.278215 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.278839 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.278844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.278848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.279001 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.279349 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.279354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.279358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.279724 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.279909 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.280256 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.280531 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:03:12.282110 15812 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:03:12.283062 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.282698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43036)
0620 15:03:12.282854 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.282932 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 15:03:12.283900 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.283607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43046)
0620 15:03:12.283756 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.283814 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 15:03:12.284101 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
I20260620 15:03:12.330418 15886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 3
I20260620 15:03:12.331763 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.331279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43062)
0620 15:03:12.331500 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.331598 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:03:12.332830 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.332419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43078)
0620 15:03:12.332629 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.332723 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260620 15:03:12.333025 15886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
I20260620 15:03:12.335289 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 1
I20260620 15:03:12.336213 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.335822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43082)
0620 15:03:12.335988 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.336071 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 15:03:12.337124 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.336819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43094)
0620 15:03:12.336957 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.337025 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:03:12.337332 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
I20260620 15:03:12.405551 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 2
I20260620 15:03:12.406682 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.406208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43098)
0620 15:03:12.406429 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.406512 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:03:12.407802 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.407308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43106)
0620 15:03:12.407558 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.407657 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 15:03:12.408104 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
W20260620 15:03:12.458750 15893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:12.459040 15893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:12.459128 15893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:12.467054 15893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:12.467166 15893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:12.467211 15893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:12.467267 15893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:12.467310 15893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:12.476246 15886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 4
I20260620 15:03:12.477344 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.476876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43110)
0620 15:03:12.477114 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.477197 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 15:03:12.477442 15893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:46349,127.2.70.61:40533,127.2.70.60:39835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab
--principal=kudu/127.2.70.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.60:39835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.70.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:12.478317 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.477984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43124)
0620 15:03:12.478150 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.478215 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:03:12.478503 15886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
I20260620 15:03:12.479038 15893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:12.480337 15893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:12.492066 15898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:12.492093 15901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:12.492172 15899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:12.493436 15893 server_base.cc:1034] running on GCE node
I20260620 15:03:12.494302 15893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:12.495769 15893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:12.497004 15893 hybrid_clock.cc:648] HybridClock initialized: now 1781967792496954 us; error 42 us; skew 500 ppm
I20260620 15:03:12.497368 15893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:12.500501 15893 init.cc:377] Logged in from keytab as kudu/127.2.70.60@KRBTEST.COM (short username kudu)
I20260620 15:03:12.502086 15893 webserver.cc:460] Webserver started at http://127.2.70.60:45343/ using document root <none> and password file <none>
I20260620 15:03:12.502815 15893 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:12.502960 15893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:12.503268 15893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:12.506471 15893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/data/instance:
uuid: "04555b79658b4e8f870f093317ad9a86"
format_stamp: "Formatted at 2026-06-20 15:03:12 on dist-test-slave-n5vv"
server_key: "9d943cedf6e062c4bf3a0abbc40a2d22"
server_key_iv: "b5dd896b8349a00b6bf9684164ac9ed2"
server_key_version: "encryptionkey@0"
I20260620 15:03:12.507726 15893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/wal/instance:
uuid: "04555b79658b4e8f870f093317ad9a86"
format_stamp: "Formatted at 2026-06-20 15:03:12 on dist-test-slave-n5vv"
server_key: "9d943cedf6e062c4bf3a0abbc40a2d22"
server_key_iv: "b5dd896b8349a00b6bf9684164ac9ed2"
server_key_version: "encryptionkey@0"
I20260620 15:03:12.514325 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 3
I20260620 15:03:12.515084 15893 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260620 15:03:12.515515 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.515029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43140)
0620 15:03:12.515283 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.515368 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:03:12.516585 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.516273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43146)
0620 15:03:12.516419 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.516485 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:03:12.516852 15812 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:39835: Network error: Client connection negotiation failed: client connection to 127.2.70.60:39835: connect: Connection refused (error 111)
I20260620 15:03:12.519141 15908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:12.520469 15893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:03:12.520663 15893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/wal
uuid: "04555b79658b4e8f870f093317ad9a86"
format_stamp: "Formatted at 2026-06-20 15:03:12 on dist-test-slave-n5vv"
server_key: "9d943cedf6e062c4bf3a0abbc40a2d22"
server_key_iv: "b5dd896b8349a00b6bf9684164ac9ed2"
server_key_version: "encryptionkey@0"
I20260620 15:03:12.520865 15893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:12.539825 15893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:12.544078 15893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:12.544377 15893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:12.565002 15893 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.60:39835
I20260620 15:03:12.565022 15959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.60:39835 every 8 connection(s)
I20260620 15:03:12.566459 15893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
I20260620 15:03:12.570148 15960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:12.573666 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 15893
I20260620 15:03:12.573858 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781967589245963-2328-0/minicluster-data/master-2/wal/instance
I20260620 15:03:12.574306 2328 external_mini_cluster.cc:1428] Setting key b7be16c7dcca48ee95102091ee200708
I20260620 15:03:12.577257 15960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:12.598289 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.581220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.2.70.62:46349)
0620 15:03:12.581501 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.581510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.582250 (+ 740us) server_negotiation.cc:407] Connection header received
0620 15:03:12.582438 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.582442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.582505 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.582588 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.583951 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.585011 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.586950 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.587600 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.587738 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.591916 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.591923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.591930 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.591974 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.595015 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.595783 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.595788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.595793 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.595970 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.596387 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.596392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.596396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.596693 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.596810 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.597608 (+ 798us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.597858 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:03:12.599458 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.581172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46349 (local address 127.0.0.1:36302)
0620 15:03:12.582079 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.582137 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:12.582321 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:12.582688 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:12.582696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:12.583512 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:12.583793 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.583801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.585181 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.585185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.586801 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.586815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.588060 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.588068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.588274 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.588878 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:12.588912 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:12.591708 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:12.595165 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.595176 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.595191 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.595639 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.596086 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.596091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.596095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.596268 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.596817 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:12.596827 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:12.597192 (+ 365us) client_negotiation.cc:769] Sending connection context
0620 15:03:12.597460 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 15:03:12.597973 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":4,"thread_start_us":480,"threads_started":1}
I20260620 15:03:12.599478 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.576168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.2.70.60:39835)
0620 15:03:12.576934 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.576947 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.576987 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:03:12.578605 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.578629 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.579053 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.579445 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.581205 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.583354 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.585322 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.586406 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.586590 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.591740 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.591749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.591762 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.591826 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.595165 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.595891 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.595898 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.595900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.596073 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.596481 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.596484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.596486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.597220 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.597519 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.597896 (+ 377us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.598092 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":569,"spinlock_wait_cycles":23936,"thread_start_us":351,"threads_started":1}
I20260620 15:03:12.601238 15960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:12.614768 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.601935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:40533 (local address 127.0.0.1:45580)
0620 15:03:12.602117 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.602137 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:12.602237 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:12.602551 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:12.602555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:12.603026 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:12.603227 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.603238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.604186 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.604190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.605017 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.605028 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.605832 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.605838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.606036 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.606595 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:12.606618 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:12.609053 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:12.612171 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.612177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.612181 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.612699 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.613043 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.613048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.613052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.613194 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.613829 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:12.613834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:12.613991 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 15:03:12.614127 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:03:12.614397 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 15:03:12.614799 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.602040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.2.70.61:40533)
0620 15:03:12.602227 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.602233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.602252 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:03:12.602364 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.602368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.602428 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.602530 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.603376 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.604054 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.605166 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.605686 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.605815 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.609221 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.609225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.609229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.609261 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.612038 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.612804 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.612808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.612810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.612934 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.613300 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.613306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.613309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.613703 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.613870 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.614218 (+ 348us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.614422 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:03:12.615759 15960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:03:12.618655 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.602879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.2.70.62:46349)
0620 15:03:12.603048 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.603053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.603071 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:03:12.603166 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.603173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.603241 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.603341 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.605371 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.606173 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.608137 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.608644 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.608738 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.612980 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.612985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.612988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.613017 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.615920 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.616690 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.616694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.616696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.616855 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.617230 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.617233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.617235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.617523 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.617751 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.618121 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.618335 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967792, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
I20260620 15:03:12.630529 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.616702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43160 (local address 127.2.70.60:39835)
0620 15:03:12.616867 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.616873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.616890 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:03:12.616938 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.616942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.617007 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.617083 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.617917 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.618896 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.620380 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.620860 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.620992 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.624750 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.624756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.624761 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.624796 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.627704 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.628559 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.628563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.628565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.628675 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.629075 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.629078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.629080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.629332 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.629437 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.629893 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.630108 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":58}
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:03:12.630529 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.616463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43160)
0620 15:03:12.616640 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.616660 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:12.616758 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:12.617140 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:12.617147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:12.617584 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:12.617778 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.617790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.619011 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.619019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.620244 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.620261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.620994 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.620999 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.621263 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.621893 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:12.621917 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:12.624559 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:12.627867 (+ 3308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.627876 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.627882 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.628433 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.628785 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.628791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.628795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.628967 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.629448 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:12.629453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:12.629630 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 15:03:12.629814 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 15:03:12.630264 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":106}
I20260620 15:03:12.636158 15960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86: Bootstrap starting.
I20260620 15:03:12.637068 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.620380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.2.70.61:40533)
0620 15:03:12.620531 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.620537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.621277 (+ 740us) server_negotiation.cc:407] Connection header received
0620 15:03:12.621437 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.621441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.621550 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.621688 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.623287 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.624341 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.625917 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.626631 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.626768 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.631829 (+ 5061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.631838 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.631843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.631883 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.634506 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.635315 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.635319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.635321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.635451 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.635858 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.635861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.635863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.636125 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.636346 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.636661 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.636827 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 15:03:12.640041 15960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:12.641098 15960 log.cc:824] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:12.642218 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:03:12.643857 15960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86: No bootstrap required, opened a new log
I20260620 15:03:12.657843 15960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.658680 15960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:12.658915 15960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04555b79658b4e8f870f093317ad9a86, State: Initialized, Role: FOLLOWER
I20260620 15:03:12.660080 15960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.663074 15973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:12.663595 15973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:03:12.664758 15960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:03:12.670002 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.647434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36324 (local address 127.2.70.62:46349)
0620 15:03:12.647623 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.647631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.648030 (+ 399us) server_negotiation.cc:407] Connection header received
0620 15:03:12.648346 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.648354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.648461 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.648551 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.651550 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.652973 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.655793 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.656746 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.656920 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.662692 (+ 5772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.662699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.662703 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.662747 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.666416 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.667281 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.667287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.667290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.667457 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.667964 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.667972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.667977 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.668369 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.668621 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.669341 (+ 720us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.669614 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 15:03:12.677964 15890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.647048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.2.70.61:40533)
0620 15:03:12.647234 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.647241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.659535 (+ 12294us) server_negotiation.cc:407] Connection header received
0620 15:03:12.660095 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.660103 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.660196 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.660281 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.662516 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.663651 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.665969 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.666631 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.666734 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.670177 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.670183 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.670187 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.670224 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.674075 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.675095 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.675101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.675105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.675283 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.675828 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.675839 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.675845 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.676379 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.676747 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.677333 (+ 586us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.677600 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:03:12.683475 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.647929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.2.70.60:39835)
0620 15:03:12.648113 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.648119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.662532 (+ 14413us) server_negotiation.cc:407] Connection header received
0620 15:03:12.664195 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.664200 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.664279 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.664366 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.666563 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.668175 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.671348 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.672440 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.672566 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.676286 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.676291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.676295 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.676325 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.679656 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.680465 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.680470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.680474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.680642 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.681035 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.681039 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.681041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.681449 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.681883 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.682125 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.682333 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 15:03:12.687379 15984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:03:12.687580 15984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:03:12.694057 15812 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 4
I20260620 15:03:12.683645 15893 master_runner.cc:186] Error getting master registration for 127.2.70.62:46349: OK, instance_id { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" instance_seqno: 1781967791549160 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:03:12.694509 15893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
I20260620 15:03:12.709566 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.694821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43170 (local address 127.2.70.60:39835)
0620 15:03:12.695015 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.695022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.695041 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:03:12.695107 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.695112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.695190 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.695293 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.696026 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.696820 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.698101 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.698530 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.698664 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.701840 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.701847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.701852 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.701893 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.705708 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.706328 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.706338 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.706340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.706539 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.706903 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.706908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.706911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.707200 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.707358 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.707913 (+ 555us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.709110 (+ 1197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:03:12.710299 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.694613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43170)
0620 15:03:12.694861 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.694879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:12.694976 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:12.695327 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:12.695332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:12.695730 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:12.695888 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.695897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.696961 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.696974 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.697947 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.697959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.698675 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.698681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.698853 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.699474 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:12.699492 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:12.701668 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:12.705834 (+ 4166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.705840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.705844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.706201 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.706651 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.706657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.706660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.706793 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.707329 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:12.707334 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:12.707504 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 15:03:12.707664 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 15:03:12.708011 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 15:03:12.714831 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: Bootstrap starting.
I20260620 15:03:12.718886 15812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:12.719947 15812 log.cc:824] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:12.722713 15812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: No bootstrap required, opened a new log
I20260620 15:03:12.730837 15812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.731149 15812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:12.731302 15812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001065394053451196f47ddb2a2d3dc0, State: Initialized, Role: FOLLOWER
I20260620 15:03:12.731945 15812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.732992 15987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:12.733511 15987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:03:12.734014 15812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:03:12.745106 15998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:03:12.745327 15998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:03:12.744748 15745 master_runner.cc:186] Error getting master registration for 127.2.70.61:40533: OK, instance_id { permanent_uuid: "83513505386d4940a523a32a76fd22a5" instance_seqno: 1781967792052567 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:03:12.758757 15745 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:03:12.777712 15886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } attempt: 5
Jun 20 15:03:12 dist-test-slave-n5vv krb5kdc[15732](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967791, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
W20260620 15:03:12.788765 15766 tablet.cc:2369] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:03:12.796568 15914 tablet.cc:2369] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:03:12.797123 15888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.778456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:39835 (local address 127.0.0.1:43180)
0620 15:03:12.778793 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:12.778815 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:12.778944 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:12.779526 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:12.779533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:12.780089 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:12.780298 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.780312 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.782116 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.782123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.783598 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:12.783613 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.784651 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.784660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:12.784848 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.785567 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:12.785590 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:12.788387 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:12.793817 (+ 5430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.793823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.793827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.794348 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.794808 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:12.794814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:12.794817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:12.794977 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:12.795704 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:12.795708 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:12.796237 (+ 529us) client_negotiation.cc:769] Sending connection context
0620 15:03:12.796403 (+ 166us) client_negotiation.cc:240] Negotiation successful
0620 15:03:12.796728 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260620 15:03:12.798202 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:12.778546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43180 (local address 127.2.70.60:39835)
0620 15:03:12.778737 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:12.778744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:12.779062 (+ 318us) server_negotiation.cc:407] Connection header received
0620 15:03:12.779128 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:12.779136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:12.779221 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:12.779319 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:12.780477 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.781733 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.783753 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:12.784484 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:12.784618 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:12.789713 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:12.789719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:12.789725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:12.789767 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:12.793645 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.794496 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.794502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.794506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.794659 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:12.795109 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:12.795114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:12.795118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:12.795468 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:12.795639 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:12.797566 (+ 1927us) server_negotiation.cc:299] Negotiation successful
0620 15:03:12.797828 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:03:12.804196 15886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5: Bootstrap starting.
I20260620 15:03:12.808310 15886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:12.809755 15886 log.cc:824] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:12.815268 15886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5: No bootstrap required, opened a new log
I20260620 15:03:12.826555 15886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.826997 15886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:12.827119 15886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83513505386d4940a523a32a76fd22a5, State: Initialized, Role: FOLLOWER
I20260620 15:03:12.827842 15886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.828863 16001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:12.829299 16001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:03:12.830425 15886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 15:03:12.857101 16012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:03:12.857347 16012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:03:12.859184 15819 master_runner.cc:263] No leader master found from master 83513505386d4940a523a32a76fd22a5
I20260620 15:03:12.859337 15819 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:03:12.862282 15973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:03:12.862609 15973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.864246 15973 leader_election.cc:288] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 001065394053451196f47ddb2a2d3dc0 (127.2.70.62:46349), 83513505386d4940a523a32a76fd22a5 (127.2.70.61:40533)
I20260620 15:03:12.866688 15861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04555b79658b4e8f870f093317ad9a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83513505386d4940a523a32a76fd22a5" is_pre_election: true
I20260620 15:03:12.867344 15861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04555b79658b4e8f870f093317ad9a86 in term 0.
I20260620 15:03:12.868155 15912 leader_election.cc:302] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04555b79658b4e8f870f093317ad9a86, 83513505386d4940a523a32a76fd22a5; no voters:
I20260620 15:03:12.865381 15787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04555b79658b4e8f870f093317ad9a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001065394053451196f47ddb2a2d3dc0" is_pre_election: true
I20260620 15:03:12.869252 15787 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04555b79658b4e8f870f093317ad9a86 in term 0.
I20260620 15:03:12.870252 15973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:03:12.870405 15973 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:03:12.870507 15973 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:12.874852 15973 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.877915 15787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04555b79658b4e8f870f093317ad9a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001065394053451196f47ddb2a2d3dc0"
I20260620 15:03:12.878171 15787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:12.881805 15787 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04555b79658b4e8f870f093317ad9a86 in term 1.
I20260620 15:03:12.882484 15909 leader_election.cc:302] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001065394053451196f47ddb2a2d3dc0, 04555b79658b4e8f870f093317ad9a86; no voters:
I20260620 15:03:12.886909 15861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04555b79658b4e8f870f093317ad9a86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83513505386d4940a523a32a76fd22a5"
I20260620 15:03:12.887220 15861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:12.891484 15861 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04555b79658b4e8f870f093317ad9a86 in term 1.
I20260620 15:03:12.892544 15973 leader_election.cc:288] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [CANDIDATE]: Term 1 election: Requested vote from peers 001065394053451196f47ddb2a2d3dc0 (127.2.70.62:46349), 83513505386d4940a523a32a76fd22a5 (127.2.70.61:40533)
I20260620 15:03:12.892977 15973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:12.893414 15973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [term 1 LEADER]: Becoming Leader. State: Replica: 04555b79658b4e8f870f093317ad9a86, State: Running, Role: LEADER
I20260620 15:03:12.894690 15973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } }
I20260620 15:03:12.909804 15973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04555b79658b4e8f870f093317ad9a86. Latest consensus state: current_term: 1 leader_uuid: "04555b79658b4e8f870f093317ad9a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:12.910279 15973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:12.915421 16016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:03:12.917780 16016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:03:12.937702 15861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [term 1 FOLLOWER]: Refusing update from remote peer 04555b79658b4e8f870f093317ad9a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:12.938875 15973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:12.946362 15787 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [term 1 FOLLOWER]: Refusing update from remote peer 04555b79658b4e8f870f093317ad9a86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:12.947332 15973 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:12.986795 15987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04555b79658b4e8f870f093317ad9a86. Latest consensus state: current_term: 1 leader_uuid: "04555b79658b4e8f870f093317ad9a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:12.987210 15987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:03:12.991451 16017 mvcc.cc:205] Tried to move back new op lower bound from 7298940079862120448 to 7298940079723532288. Current Snapshot: MvccSnapshot[applied={T|T < 7298940079862120448}]
I20260620 15:03:12.995316 16019 mvcc.cc:205] Tried to move back new op lower bound from 7298940079862120448 to 7298940079723532288. Current Snapshot: MvccSnapshot[applied={T|T < 7298940079862120448}]
I20260620 15:03:13.002596 15987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04555b79658b4e8f870f093317ad9a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:13.002892 15987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:03:13.002760 16001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04555b79658b4e8f870f093317ad9a86. Latest consensus state: current_term: 1 leader_uuid: "04555b79658b4e8f870f093317ad9a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:13.003091 16001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:03:13.007745 16014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04555b79658b4e8f870f093317ad9a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:13.008191 16014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:13.013794 15973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "04555b79658b4e8f870f093317ad9a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:13.014506 15973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:13.015337 16018 mvcc.cc:205] Tried to move back new op lower bound from 7298940079862120448 to 7298940079723532288. Current Snapshot: MvccSnapshot[applied={T|T < 7298940079862120448}]
I20260620 15:03:13.020754 16016 catalog_manager.cc:1347] Generated new cluster ID: f810d742557c48dfbe4848418edddcfc
I20260620 15:03:13.020862 16016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:03:13.030202 16001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "04555b79658b4e8f870f093317ad9a86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001065394053451196f47ddb2a2d3dc0" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46349 } } peers { permanent_uuid: "83513505386d4940a523a32a76fd22a5" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 40533 } } peers { permanent_uuid: "04555b79658b4e8f870f093317ad9a86" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 39835 } } }
I20260620 15:03:13.030471 16001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:03:13.071213 16016 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:03:13.073156 16016 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:03:13.096520 16016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04555b79658b4e8f870f093317ad9a86: Generated new TSK 0
I20260620 15:03:13.097663 16016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:03:13.747263 15998 catalog_manager.cc:1259] Loaded cluster ID: f810d742557c48dfbe4848418edddcfc
I20260620 15:03:13.747395 15998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: loading cluster ID for follower catalog manager: success
I20260620 15:03:13.750022 15998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: acquiring CA information for follower catalog manager: success
I20260620 15:03:13.751854 15998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 001065394053451196f47ddb2a2d3dc0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:03:13.859375 16012 catalog_manager.cc:1259] Loaded cluster ID: f810d742557c48dfbe4848418edddcfc
I20260620 15:03:13.859486 16012 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5: loading cluster ID for follower catalog manager: success
I20260620 15:03:13.862001 16012 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5: acquiring CA information for follower catalog manager: success
I20260620 15:03:13.863739 16012 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 83513505386d4940a523a32a76fd22a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:03:14.254045 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 15745
I20260620 15:03:14.255470 15807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:14.368795 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 15745
I20260620 15:03:14.390259 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 15819
I20260620 15:03:14.391449 15881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:14.498762 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 15819
I20260620 15:03:14.521400 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 15893
I20260620 15:03:14.522547 15955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:14.635260 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 15893
2026-06-20T15:03:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4891 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:03:14 dist-test-slave-n5vv krb5kdc[16036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:03:14 dist-test-slave-n5vv krb5kdc[16036](info): set up 2 sockets
Jun 20 15:03:14 dist-test-slave-n5vv krb5kdc[16036](info): commencing operation
krb5kdc: starting...
W20260620 15:03:15.806000 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:03:15 dist-test-slave-n5vv krb5kdc[16036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:03:16.041973 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43111
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:40907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:16.373593 16049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:16.373941 16049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:16.374106 16049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:16.381194 16049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:16.381346 16049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:16.381475 16049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:16.381592 16049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:16.381686 16049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:16.391542 16049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:40907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:16.392922 16049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:16.394284 16049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:16.406368 16054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:16.406440 16057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:16.406420 16055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:16.407192 16049 server_base.cc:1034] running on GCE node
I20260620 15:03:16.408008 16049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:16.409560 16049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:16.410773 16049 hybrid_clock.cc:648] HybridClock initialized: now 1781967796410728 us; error 38 us; skew 500 ppm
I20260620 15:03:16.411132 16049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:16 dist-test-slave-n5vv krb5kdc[16036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967796, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:16.421288 16049 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:03:16.422919 16049 webserver.cc:460] Webserver started at http://127.2.70.62:35045/ using document root <none> and password file <none>
I20260620 15:03:16.423683 16049 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:16.423830 16049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:16.424141 16049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:16.427210 16049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "599c35524592456f933f3b41141c7eac"
format_stamp: "Formatted at 2026-06-20 15:03:16 on dist-test-slave-n5vv"
server_key: "a12a691aef7171f44c72cefd1941287a"
server_key_iv: "e46e7d4b81ae7be33889f3a72d25155f"
server_key_version: "encryptionkey@0"
I20260620 15:03:16.427989 16049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "599c35524592456f933f3b41141c7eac"
format_stamp: "Formatted at 2026-06-20 15:03:16 on dist-test-slave-n5vv"
server_key: "a12a691aef7171f44c72cefd1941287a"
server_key_iv: "e46e7d4b81ae7be33889f3a72d25155f"
server_key_version: "encryptionkey@0"
I20260620 15:03:16.433256 16049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:03:16.436758 16064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:16.438151 16049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:16.438344 16049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "599c35524592456f933f3b41141c7eac"
format_stamp: "Formatted at 2026-06-20 15:03:16 on dist-test-slave-n5vv"
server_key: "a12a691aef7171f44c72cefd1941287a"
server_key_iv: "e46e7d4b81ae7be33889f3a72d25155f"
server_key_version: "encryptionkey@0"
I20260620 15:03:16.438560 16049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:16.454731 16049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:16.459719 16049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:16.460027 16049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:16.479099 16049 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:43111
I20260620 15:03:16.479126 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:43111 every 8 connection(s)
I20260620 15:03:16.480528 16049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:03:16.484311 16116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:16.484966 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 16049
I20260620 15:03:16.485216 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:03:16.486119 2328 external_mini_cluster.cc:1428] Setting key 8b004330c55b5bde6658e4d7336b0250
I20260620 15:03:16.498267 16116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac: Bootstrap starting.
Jun 20 15:03:16 dist-test-slave-n5vv krb5kdc[16036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:16.502133 16116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:16.503166 16116 log.cc:824] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:16.506539 16116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac: No bootstrap required, opened a new log
I20260620 15:03:16.512295 16119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:16.487946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.2.70.62:43111)
0620 15:03:16.488726 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:16.488738 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:16.488778 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:03:16.490700 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:16.490721 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:16.491109 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:16.491429 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:16.493278 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:16.494651 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:16.496567 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:16.497597 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:16.497728 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:16.502623 (+ 4895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:16.502630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:16.502644 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:16.502678 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:16.506433 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:16.507553 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:16.507561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:16.507569 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:16.507775 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:16.508262 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:16.508267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:16.508270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:16.508986 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:16.509295 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:16.510254 (+ 959us) server_negotiation.cc:299] Negotiation successful
0620 15:03:16.510842 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":352,"threads_started":1}
I20260620 15:03:16.517665 16116 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599c35524592456f933f3b41141c7eac" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43111 } }
I20260620 15:03:16.518128 16116 raft_consensus.cc:374] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:16.518224 16116 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599c35524592456f933f3b41141c7eac, State: Initialized, Role: FOLLOWER
I20260620 15:03:16.518383 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:03:16.518805 16116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599c35524592456f933f3b41141c7eac" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43111 } }
I20260620 15:03:16.519035 16116 raft_consensus.cc:388] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:03:16.519160 16116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:03:16.519330 16116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:16.521831 16116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599c35524592456f933f3b41141c7eac" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43111 } }
I20260620 15:03:16.522305 16116 leader_election.cc:302] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 599c35524592456f933f3b41141c7eac; no voters:
I20260620 15:03:16.522869 16116 leader_election.cc:288] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:03:16.523001 16121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:16.523687 16121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [term 1 LEADER]: Becoming Leader. State: Replica: 599c35524592456f933f3b41141c7eac, State: Running, Role: LEADER
I20260620 15:03:16.524268 16121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599c35524592456f933f3b41141c7eac" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43111 } }
I20260620 15:03:16.524868 16116 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:03:16.527040 16123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599c35524592456f933f3b41141c7eac. Latest consensus state: current_term: 1 leader_uuid: "599c35524592456f933f3b41141c7eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599c35524592456f933f3b41141c7eac" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43111 } } }
I20260620 15:03:16.527488 16123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:16.528506 16122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "599c35524592456f933f3b41141c7eac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599c35524592456f933f3b41141c7eac" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43111 } } }
I20260620 15:03:16.528764 16122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:16.529275 16128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:03:16.533871 16128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:03:16.541673 16128 catalog_manager.cc:1347] Generated new cluster ID: 3b7f70ab35db4d1587a274f52eeeb008
I20260620 15:03:16.541783 16128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:03:16.550980 16128 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:03:16.552774 16128 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:03:16.563933 16128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 599c35524592456f933f3b41141c7eac: Generated new TSK 0
I20260620 15:03:16.564936 16128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:03:16.579547 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 16049
I20260620 15:03:16.580785 16111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:16.671142 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 16049
2026-06-20T15:03:16Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2031 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:03:16 dist-test-slave-n5vv krb5kdc[16144](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:03:16 dist-test-slave-n5vv krb5kdc[16144](info): set up 2 sockets
Jun 20 15:03:16 dist-test-slave-n5vv krb5kdc[16144](info): commencing operation
krb5kdc: starting...
W20260620 15:03:17.774266 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:03:17 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:03:18.007249 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34541
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:18.334380 16157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:18.334681 16157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:18.334774 16157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:18.342070 16157 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:18.342192 16157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:18.342238 16157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:18.342296 16157 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:18.342340 16157 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:18.352358 16157 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:18.353705 16157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:18.354969 16157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:18.366825 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:18.367004 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:18.366861 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:18.368160 16157 server_base.cc:1034] running on GCE node
I20260620 15:03:18.368973 16157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:18.370502 16157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:18.371706 16157 hybrid_clock.cc:648] HybridClock initialized: now 1781967798371642 us; error 51 us; skew 500 ppm
I20260620 15:03:18.372044 16157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:18 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:18.375109 16157 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:03:18.376704 16157 webserver.cc:460] Webserver started at http://127.2.70.62:37113/ using document root <none> and password file <none>
I20260620 15:03:18.377379 16157 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:18.377544 16157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:18.377848 16157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:18.381246 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "d4210f48d96e43d2b11656eb54a3e6b2"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "8c55fbe778456632dad452ae02c6d3e7"
server_key_iv: "c44757842988c670d54ec077163730ca"
server_key_version: "encryptionkey@0"
I20260620 15:03:18.382061 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "d4210f48d96e43d2b11656eb54a3e6b2"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "8c55fbe778456632dad452ae02c6d3e7"
server_key_iv: "c44757842988c670d54ec077163730ca"
server_key_version: "encryptionkey@0"
I20260620 15:03:18.387180 16157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:03:18.390681 16172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:18.391996 16157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:03:18.392182 16157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "d4210f48d96e43d2b11656eb54a3e6b2"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "8c55fbe778456632dad452ae02c6d3e7"
server_key_iv: "c44757842988c670d54ec077163730ca"
server_key_version: "encryptionkey@0"
I20260620 15:03:18.392385 16157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:18.411085 16157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:18.415037 16157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:18.415325 16157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:18.433787 16157 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:34541
I20260620 15:03:18.433810 16223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:34541 every 8 connection(s)
I20260620 15:03:18.435191 16157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:03:18.439078 16224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:18.439333 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 16157
I20260620 15:03:18.439544 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:03:18.440408 2328 external_mini_cluster.cc:1428] Setting key a67fd1cd526f4c18f0fe788428ecf9cd
I20260620 15:03:18.451839 16224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Bootstrap starting.
Jun 20 15:03:18 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:18.455470 16224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:18.456555 16224 log.cc:824] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:18.459751 16224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: No bootstrap required, opened a new log
I20260620 15:03:18.466156 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:18.442602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.2.70.62:34541)
0620 15:03:18.443353 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:18.443364 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:18.443408 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:03:18.445601 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:18.445625 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:18.446031 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:18.446473 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:18.447944 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:18.449285 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:18.450790 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:18.451492 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:18.451700 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:18.456458 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:18.456466 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:18.456483 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:18.456524 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:18.460344 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:18.461589 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:18.461599 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:18.461609 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:18.461827 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:18.462434 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:18.462439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:18.462443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:18.463273 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:18.463606 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:18.464048 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 15:03:18.464585 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":310,"threads_started":1}
I20260620 15:03:18.470314 16224 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:18.470681 16224 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:18.470758 16224 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4210f48d96e43d2b11656eb54a3e6b2, State: Initialized, Role: FOLLOWER
I20260620 15:03:18.471251 16224 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:18.471444 16224 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:03:18.471540 16224 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:03:18.471661 16224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:18.474066 16224 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:18.474504 16224 leader_election.cc:302] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4210f48d96e43d2b11656eb54a3e6b2; no voters:
I20260620 15:03:18.475037 16224 leader_election.cc:288] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:03:18.475193 16229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:18.475863 16229 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 1 LEADER]: Becoming Leader. State: Replica: d4210f48d96e43d2b11656eb54a3e6b2, State: Running, Role: LEADER
I20260620 15:03:18.476352 16229 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:18.477157 16224 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:03:18.480676 16231 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4210f48d96e43d2b11656eb54a3e6b2. Latest consensus state: current_term: 1 leader_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } } }
I20260620 15:03:18.480813 16230 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } } }
I20260620 15:03:18.481220 16231 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:18.481449 16230 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:18.484782 16238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:03:18.490406 16238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:03:18.504590 16238 catalog_manager.cc:1347] Generated new cluster ID: 000a03048eaa498b8d82da5a52a11bf2
I20260620 15:03:18.504733 16238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:03:18.533593 16238 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:03:18.534996 16238 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:03:18.541424 16238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Generated new TSK 0
I20260620 15:03:18.542326 16238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:03:18.609880 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34541
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:18.944145 16250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:18.944432 16250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:18.944564 16250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:18.951797 16250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:18.951915 16250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:18.952121 16250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:03:18.962067 16250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:18.963544 16250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:18.965019 16250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:18.979628 16255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:18.979583 16258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:18.979667 16256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:18.980821 16250 server_base.cc:1034] running on GCE node
I20260620 15:03:18.981451 16250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:18.982863 16250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:18.984138 16250 hybrid_clock.cc:648] HybridClock initialized: now 1781967798984089 us; error 75 us; skew 500 ppm
I20260620 15:03:18.984495 16250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:18 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:18.987838 16250 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:03:18.989254 16250 webserver.cc:460] Webserver started at http://127.2.70.1:34827/ using document root <none> and password file <none>
I20260620 15:03:18.989997 16250 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:18.990100 16250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:18.990334 16250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:18.993276 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "b10d37c6bd6745659a9a4a7fd97208d9"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "248316872791e47636323555505dfa52"
server_key_iv: "c79bf2a7b493afd5e6912418f322153d"
server_key_version: "encryptionkey@0"
I20260620 15:03:18.994174 16250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "b10d37c6bd6745659a9a4a7fd97208d9"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "248316872791e47636323555505dfa52"
server_key_iv: "c79bf2a7b493afd5e6912418f322153d"
server_key_version: "encryptionkey@0"
I20260620 15:03:18.999643 16250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:03:19.003350 16265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:19.004904 16250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:03:19.005088 16250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "b10d37c6bd6745659a9a4a7fd97208d9"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "248316872791e47636323555505dfa52"
server_key_iv: "c79bf2a7b493afd5e6912418f322153d"
server_key_version: "encryptionkey@0"
I20260620 15:03:19.005295 16250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:19.022614 16250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:19.026854 16250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:19.027154 16250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:19.028131 16250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:19.029560 16250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:19.029650 16250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:19.029800 16250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:19.029865 16250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:19.063508 16250 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:33045
I20260620 15:03:19.063544 16377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:33045 every 8 connection(s)
I20260620 15:03:19.064812 16250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:03:19.075212 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 16250
I20260620 15:03:19.075433 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:03:19.076198 2328 external_mini_cluster.cc:1428] Setting key 0ea93cad0dbbce5c1c181f7f7a77d078
Jun 20 15:03:19 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967798, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:19.086879 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:19.067410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:58913 (local address 127.2.70.62:34541)
0620 15:03:19.067867 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:19.067874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:19.068882 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 15:03:19.070778 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:19.070783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:19.070850 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:19.070946 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:19.072500 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:19.073234 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:19.074802 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:19.075444 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:19.075600 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:19.079764 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:19.079774 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:19.079783 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:19.079830 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:19.083685 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:19.084439 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:19.084448 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:19.084455 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:19.084612 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:19.085018 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:19.085026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:19.085032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:19.085450 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:19.085641 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:19.086311 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 15:03:19.086582 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":216,"threads_started":1}
I20260620 15:03:19.088208 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:19.067996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:58913)
0620 15:03:19.068720 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:19.068758 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:19.070525 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:19.071273 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:19.071283 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:19.071793 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:19.072351 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:19.072367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:19.073383 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:19.073387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:19.074642 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:19.074654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:19.075698 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:19.075702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:19.075893 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:19.076565 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:19.076586 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:19.079529 (+ 2943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:19.083863 (+ 4334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:19.083870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:19.083884 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:19.084298 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:19.084721 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:19.084724 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:19.084726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:19.084904 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:19.085608 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:19.085614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:19.085929 (+ 315us) client_negotiation.cc:769] Sending connection context
0620 15:03:19.086158 (+ 229us) client_negotiation.cc:240] Negotiation successful
0620 15:03:19.086744 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":272,"threads_started":1}
I20260620 15:03:19.089823 16378 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34541
I20260620 15:03:19.090122 16378 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:19.090840 16378 heartbeater.cc:511] Master 127.2.70.62:34541 requested a full tablet report, sending...
I20260620 15:03:19.092880 16189 ts_manager.cc:194] Registered new tserver with Master: b10d37c6bd6745659a9a4a7fd97208d9 (127.2.70.1:33045)
I20260620 15:03:19.094372 16189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:58913
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:03:19.132715 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34541
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:19.467234 16384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:19.467574 16384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:19.467753 16384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:19.475061 16384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:19.475219 16384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:19.475391 16384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:03:19.485186 16384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:19.486575 16384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:19.487941 16384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:19.501638 16390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:19.501854 16392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:19.502002 16389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:19.502339 16384 server_base.cc:1034] running on GCE node
I20260620 15:03:19.503039 16384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:19.503997 16384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:19.505241 16384 hybrid_clock.cc:648] HybridClock initialized: now 1781967799505185 us; error 34 us; skew 500 ppm
I20260620 15:03:19.505646 16384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:19 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967799, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:19.508839 16384 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:03:19.510480 16384 webserver.cc:460] Webserver started at http://127.2.70.2:37347/ using document root <none> and password file <none>
I20260620 15:03:19.511142 16384 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:19.511240 16384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:19.511474 16384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:19.514377 16384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "792d15b325a043c68dc06417cf438b11"
format_stamp: "Formatted at 2026-06-20 15:03:19 on dist-test-slave-n5vv"
server_key: "c833e35d9535b10c7bec484b99cb1634"
server_key_iv: "3029a71d462228c6eb02c096765963c4"
server_key_version: "encryptionkey@0"
I20260620 15:03:19.515089 16384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "792d15b325a043c68dc06417cf438b11"
format_stamp: "Formatted at 2026-06-20 15:03:19 on dist-test-slave-n5vv"
server_key: "c833e35d9535b10c7bec484b99cb1634"
server_key_iv: "3029a71d462228c6eb02c096765963c4"
server_key_version: "encryptionkey@0"
I20260620 15:03:19.520251 16384 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:03:19.523702 16399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:19.525041 16384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:03:19.525218 16384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "792d15b325a043c68dc06417cf438b11"
format_stamp: "Formatted at 2026-06-20 15:03:19 on dist-test-slave-n5vv"
server_key: "c833e35d9535b10c7bec484b99cb1634"
server_key_iv: "3029a71d462228c6eb02c096765963c4"
server_key_version: "encryptionkey@0"
I20260620 15:03:19.525413 16384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:19.538502 16384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:19.543601 16384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:19.543896 16384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:19.544879 16384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:19.546206 16384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:19.546281 16384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:19.546458 16384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:19.546512 16384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:19.578363 16384 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:42041
I20260620 15:03:19.578406 16511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:42041 every 8 connection(s)
I20260620 15:03:19.579663 16384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:03:19.585335 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 16384
I20260620 15:03:19.585546 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:03:19.586167 2328 external_mini_cluster.cc:1428] Setting key e219c977bf1f9b2651c66261b3e13c1e
Jun 20 15:03:19 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967799, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:19.603248 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:19.582213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:34045 (local address 127.2.70.62:34541)
0620 15:03:19.582392 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:19.582397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:19.583637 (+ 1240us) server_negotiation.cc:407] Connection header received
0620 15:03:19.585823 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:19.585827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:19.585899 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:19.585987 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:19.588119 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:19.589470 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:19.591720 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:19.592154 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:19.592291 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:19.596134 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:19.596142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:19.596150 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:19.596193 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:19.599873 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:19.600633 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:19.600641 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:19.600647 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:19.600811 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:19.601213 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:19.601221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:19.601227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:19.601648 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:19.601794 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:19.602524 (+ 730us) server_negotiation.cc:299] Negotiation successful
0620 15:03:19.602793 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:03:19.603964 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:19.582741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:34045)
0620 15:03:19.583462 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:19.583500 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:19.585464 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:19.586269 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:19.586283 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:19.587062 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:19.587853 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:19.587880 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:19.589749 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:19.589758 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:19.591570 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:19.591582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:19.592383 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:19.592389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:19.592639 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:19.593351 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:19.593374 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:19.595883 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:19.600042 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:19.600051 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:19.600068 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:19.600499 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:19.600916 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:19.600919 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:19.600921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:19.601107 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:19.601780 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:19.601786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:19.602129 (+ 343us) client_negotiation.cc:769] Sending connection context
0620 15:03:19.602356 (+ 227us) client_negotiation.cc:240] Negotiation successful
0620 15:03:19.602791 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":289,"threads_started":1}
I20260620 15:03:19.605623 16512 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34541
I20260620 15:03:19.605947 16512 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:19.606698 16512 heartbeater.cc:511] Master 127.2.70.62:34541 requested a full tablet report, sending...
I20260620 15:03:19.608415 16189 ts_manager.cc:194] Registered new tserver with Master: 792d15b325a043c68dc06417cf438b11 (127.2.70.2:42041)
I20260620 15:03:19.609373 16189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:34045
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:03:19.641822 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34541
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:19.970124 16517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:19.970413 16517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:19.970607 16517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:19.978027 16517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:19.978163 16517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:19.978348 16517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:03:19.988337 16517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:19.989748 16517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:19.991065 16517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:20.004505 16522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:20.004555 16525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:20.004614 16523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:20.005594 16517 server_base.cc:1034] running on GCE node
I20260620 15:03:20.006457 16517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:20.007380 16517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:20.008646 16517 hybrid_clock.cc:648] HybridClock initialized: now 1781967800008595 us; error 58 us; skew 500 ppm
I20260620 15:03:20.008993 16517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:20 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:20.012208 16517 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:03:20.013798 16517 webserver.cc:460] Webserver started at http://127.2.70.3:40015/ using document root <none> and password file <none>
I20260620 15:03:20.014490 16517 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:20.014603 16517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:20.014860 16517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:20.017906 16517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "401c08ee81bf452097fc53c65dd88a4f"
format_stamp: "Formatted at 2026-06-20 15:03:20 on dist-test-slave-n5vv"
server_key: "bbbb344b2895750e2783ab940ba9eb32"
server_key_iv: "590755bc09a6271a5a7e58e18fc10523"
server_key_version: "encryptionkey@0"
I20260620 15:03:20.018666 16517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "401c08ee81bf452097fc53c65dd88a4f"
format_stamp: "Formatted at 2026-06-20 15:03:20 on dist-test-slave-n5vv"
server_key: "bbbb344b2895750e2783ab940ba9eb32"
server_key_iv: "590755bc09a6271a5a7e58e18fc10523"
server_key_version: "encryptionkey@0"
I20260620 15:03:20.023849 16517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:03:20.027195 16532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:20.028469 16517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:03:20.028687 16517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "401c08ee81bf452097fc53c65dd88a4f"
format_stamp: "Formatted at 2026-06-20 15:03:20 on dist-test-slave-n5vv"
server_key: "bbbb344b2895750e2783ab940ba9eb32"
server_key_iv: "590755bc09a6271a5a7e58e18fc10523"
server_key_version: "encryptionkey@0"
I20260620 15:03:20.028898 16517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:20.065164 16517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:20.069602 16517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:20.069890 16517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:20.070806 16517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:20.072086 16517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:20.072190 16517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:20.072341 16517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:20.072414 16517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:20.097591 16378 heartbeater.cc:503] Master 127.2.70.62:34541 was elected leader, sending a full tablet report...
I20260620 15:03:20.106498 16517 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:34859
I20260620 15:03:20.106590 16644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:34859 every 8 connection(s)
I20260620 15:03:20.107785 16517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:03:20.113721 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 16517
I20260620 15:03:20.113912 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:03:20.114351 2328 external_mini_cluster.cc:1428] Setting key 91911e6102bf5f240da981be2183c118
Jun 20 15:03:20 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967800, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:20.127621 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.110409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:51621 (local address 127.2.70.62:34541)
0620 15:03:20.110851 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.110857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.111944 (+ 1087us) server_negotiation.cc:407] Connection header received
0620 15:03:20.113916 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.113922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.114015 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.114161 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:20.115916 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.116801 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.118382 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.118789 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.118929 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.122226 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:20.122231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:20.122235 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:20.122268 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:20.124627 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.125323 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.125328 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.125334 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.125458 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.125897 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.125902 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.125906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.126173 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:20.126334 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.127014 (+ 680us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.127250 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260620 15:03:20.128821 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.111072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:51621)
0620 15:03:20.111782 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:20.111819 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:20.113645 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:20.114392 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:20.114401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:20.115090 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:20.115749 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.115770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.116949 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.116954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.118239 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.118252 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.119023 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.119029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.119250 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.119851 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:20.119878 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:20.122020 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:20.124769 (+ 2749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:20.124778 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:20.124794 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:20.125197 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:20.125587 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:20.125600 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:20.125604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:20.125778 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:20.126308 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:20.126316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:20.126646 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 15:03:20.126910 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 15:03:20.127422 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":261,"threads_started":1}
I20260620 15:03:20.130225 16645 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34541
I20260620 15:03:20.130546 16645 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:20.131234 16645 heartbeater.cc:511] Master 127.2.70.62:34541 requested a full tablet report, sending...
I20260620 15:03:20.132568 16189 ts_manager.cc:194] Registered new tserver with Master: 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859)
I20260620 15:03:20.133322 16189 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:51621
I20260620 15:03:20.141765 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:03:20.158236 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.146358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.2.70.62:34541)
0620 15:03:20.146525 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.146530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.147024 (+ 494us) server_negotiation.cc:407] Connection header received
0620 15:03:20.147144 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.147148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.147207 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.147285 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:20.148553 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.149272 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.150645 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.151060 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.151169 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.153644 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:20.153649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:20.153653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:20.153688 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:20.156003 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.156619 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.156622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.156624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.156740 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.157036 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.157039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.157041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.157351 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:20.157597 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.157839 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.158000 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:03:20.162029 16189 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:03:20.168924 16189 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:03:20.201674 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.180872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:42041 (local address 127.0.0.1:37890)
0620 15:03:20.186529 (+ 5657us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:20.186574 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:20.186769 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:20.189336 (+ 2567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:20.189348 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:20.189400 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:20.189952 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.189970 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.193829 (+ 3859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.193835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.196401 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.196424 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.200591 (+ 4167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.200598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.200800 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.200857 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 15:03:20.201033 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 15:03:20.201248 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5405,"thread_start_us":5268,"threads_started":1}
I20260620 15:03:20.201671 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.181410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.2.70.2:42041)
0620 15:03:20.185943 (+ 4533us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.185956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.186739 (+ 783us) server_negotiation.cc:407] Connection header received
0620 15:03:20.187018 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.187030 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.187389 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.188960 (+ 1571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:20.190239 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.193665 (+ 3426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.196605 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.200327 (+ 3722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.200524 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.200743 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.201080 (+ 337us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.201272 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4379,"thread_start_us":4265,"threads_started":1}
I20260620 15:03:20.202946 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.181245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33045 (local address 127.0.0.1:58086)
0620 15:03:20.193392 (+ 12147us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:20.193411 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:20.193829 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:20.194854 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:20.194859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:20.194883 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:20.195312 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.195324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.197392 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.197397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.199780 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.199796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.202138 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.202148 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.202345 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.202379 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:03:20.202479 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 15:03:20.202620 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12021,"thread_start_us":12331,"threads_started":1}
I20260620 15:03:20.203048 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.181961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58086 (local address 127.2.70.1:33045)
0620 15:03:20.182804 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.182817 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.193539 (+ 10722us) server_negotiation.cc:407] Connection header received
0620 15:03:20.194067 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.194078 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.194355 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.194541 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:20.195509 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.197165 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.199961 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.201622 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.201758 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.201979 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.202543 (+ 564us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.202706 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":360,"threads_started":1}
I20260620 15:03:20.206156 16447 tablet_service.cc:1467] Processing CreateTablet for tablet 185b1c37dc1f4a0ead36d26976fd54cf (DEFAULT_TABLE table=test-table [id=ffcc81e61f6644e087738eecc7f52c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:20.206610 16313 tablet_service.cc:1467] Processing CreateTablet for tablet 185b1c37dc1f4a0ead36d26976fd54cf (DEFAULT_TABLE table=test-table [id=ffcc81e61f6644e087738eecc7f52c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:20.207468 16447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185b1c37dc1f4a0ead36d26976fd54cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:20.207985 16313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185b1c37dc1f4a0ead36d26976fd54cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:20.208592 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.187244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34859 (local address 127.0.0.1:49122)
0620 15:03:20.198864 (+ 11620us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:20.198890 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:20.199033 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:20.200604 (+ 1571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:20.200609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:20.200636 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:20.201126 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.201139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.203903 (+ 2764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.203909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.205996 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.206011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.207770 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.207778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.207953 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.207986 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 15:03:20.208104 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 15:03:20.208231 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11445,"mutex_wait_us":1,"thread_start_us":11902,"threads_started":1}
I20260620 15:03:20.208770 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.187302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.2.70.3:34859)
0620 15:03:20.199850 (+ 12548us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.199856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.199914 (+ 58us) server_negotiation.cc:407] Connection header received
0620 15:03:20.200008 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.200018 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.200312 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.201330 (+ 1018us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:20.201655 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.203753 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.206167 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.207561 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.207708 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.207920 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.208204 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.208364 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12386,"thread_start_us":12238,"threads_started":1}
I20260620 15:03:20.212059 16580 tablet_service.cc:1467] Processing CreateTablet for tablet 185b1c37dc1f4a0ead36d26976fd54cf (DEFAULT_TABLE table=test-table [id=ffcc81e61f6644e087738eecc7f52c9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:20.213320 16580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 185b1c37dc1f4a0ead36d26976fd54cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:20.222404 16664 tablet_bootstrap.cc:492] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Bootstrap starting.
I20260620 15:03:20.224884 16665 tablet_bootstrap.cc:492] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: Bootstrap starting.
I20260620 15:03:20.226190 16666 tablet_bootstrap.cc:492] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Bootstrap starting.
I20260620 15:03:20.226388 16664 tablet_bootstrap.cc:654] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:20.227967 16664 log.cc:824] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:20.228338 16665 tablet_bootstrap.cc:654] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:20.229682 16665 log.cc:824] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:20.229693 16666 tablet_bootstrap.cc:654] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:20.230885 16666 log.cc:824] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:20.232059 16664 tablet_bootstrap.cc:492] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: No bootstrap required, opened a new log
I20260620 15:03:20.232393 16664 ts_tablet_manager.cc:1397] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260620 15:03:20.232695 16665 tablet_bootstrap.cc:492] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: No bootstrap required, opened a new log
I20260620 15:03:20.232951 16665 ts_tablet_manager.cc:1397] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 15:03:20.234203 16666 tablet_bootstrap.cc:492] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: No bootstrap required, opened a new log
I20260620 15:03:20.234465 16666 ts_tablet_manager.cc:1397] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260620 15:03:20.240288 16665 raft_consensus.cc:348] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.240695 16665 raft_consensus.cc:374] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:20.240829 16665 raft_consensus.cc:729] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401c08ee81bf452097fc53c65dd88a4f, State: Initialized, Role: FOLLOWER
I20260620 15:03:20.241544 16665 consensus_queue.cc:260] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.242660 16645 heartbeater.cc:503] Master 127.2.70.62:34541 was elected leader, sending a full tablet report...
I20260620 15:03:20.243438 16665 ts_tablet_manager.cc:1428] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: Time spent starting tablet: real 0.010s user 0.000s sys 0.010s
I20260620 15:03:20.243374 16664 raft_consensus.cc:348] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.243875 16664 raft_consensus.cc:374] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:20.243966 16664 raft_consensus.cc:729] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792d15b325a043c68dc06417cf438b11, State: Initialized, Role: FOLLOWER
I20260620 15:03:20.244838 16664 consensus_queue.cc:260] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.246057 16512 heartbeater.cc:503] Master 127.2.70.62:34541 was elected leader, sending a full tablet report...
I20260620 15:03:20.246022 16666 raft_consensus.cc:348] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.246448 16666 raft_consensus.cc:374] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:20.246542 16666 raft_consensus.cc:729] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10d37c6bd6745659a9a4a7fd97208d9, State: Initialized, Role: FOLLOWER
I20260620 15:03:20.246733 16664 ts_tablet_manager.cc:1428] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 15:03:20.247128 16666 consensus_queue.cc:260] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.248858 16666 ts_tablet_manager.cc:1428] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 15:03:20.259035 16671 raft_consensus.cc:482] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:03:20.259335 16671 raft_consensus.cc:504] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.261766 16671 leader_election.cc:288] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b10d37c6bd6745659a9a4a7fd97208d9 (127.2.70.1:33045), 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859)
I20260620 15:03:20.270203 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.261743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:36155 (local address 127.2.70.3:34859)
0620 15:03:20.261926 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.261932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.262556 (+ 624us) server_negotiation.cc:407] Connection header received
0620 15:03:20.262790 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.262795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.262894 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.262983 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:20.263630 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.264750 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.267787 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.269047 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.269171 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.269274 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.269803 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.269940 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 15:03:20.270195 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.261406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:44265 (local address 127.2.70.1:33045)
0620 15:03:20.261655 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.261662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.262801 (+ 1139us) server_negotiation.cc:407] Connection header received
0620 15:03:20.262954 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.262962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.263033 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.263115 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:20.263775 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.264818 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.267786 (+ 2968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.269027 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.269171 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.269274 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.269803 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.269937 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 15:03:20.270464 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.261319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33045 (local address 127.2.70.2:44265)
0620 15:03:20.262722 (+ 1403us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:20.262736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:20.262836 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:20.263148 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:20.263152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:20.263190 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:20.263638 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.263648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.264930 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.264936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.267619 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.267635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.269318 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.269323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.269558 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.269664 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 15:03:20.269763 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:03:20.269988 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1277,"mutex_wait_us":345,"thread_start_us":1078,"threads_started":1}
I20260620 15:03:20.270498 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.261304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34859 (local address 127.2.70.2:36155)
0620 15:03:20.262446 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:20.262464 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:20.262646 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:20.263005 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:20.263008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:20.263027 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:20.263470 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.263488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.264896 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.264903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.267619 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:20.267634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.269306 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.269313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:20.269563 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.269664 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 15:03:20.269763 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:03:20.269988 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1005,"mutex_wait_us":269,"thread_start_us":794,"threads_started":1}
I20260620 15:03:20.271683 16600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185b1c37dc1f4a0ead36d26976fd54cf" candidate_uuid: "792d15b325a043c68dc06417cf438b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401c08ee81bf452097fc53c65dd88a4f" is_pre_election: true
I20260620 15:03:20.271683 16333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185b1c37dc1f4a0ead36d26976fd54cf" candidate_uuid: "792d15b325a043c68dc06417cf438b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" is_pre_election: true
I20260620 15:03:20.272269 16333 raft_consensus.cc:2454] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 792d15b325a043c68dc06417cf438b11 in term 0.
I20260620 15:03:20.272264 16600 raft_consensus.cc:2454] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 792d15b325a043c68dc06417cf438b11 in term 0.
I20260620 15:03:20.272985 16403 leader_election.cc:302] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 792d15b325a043c68dc06417cf438b11, b10d37c6bd6745659a9a4a7fd97208d9; no voters:
I20260620 15:03:20.273424 16671 raft_consensus.cc:2781] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:03:20.273626 16671 raft_consensus.cc:482] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:03:20.273766 16671 raft_consensus.cc:3037] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:20.275902 16671 raft_consensus.cc:504] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.276491 16671 leader_election.cc:288] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [CANDIDATE]: Term 1 election: Requested vote from peers b10d37c6bd6745659a9a4a7fd97208d9 (127.2.70.1:33045), 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859)
I20260620 15:03:20.276887 16333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185b1c37dc1f4a0ead36d26976fd54cf" candidate_uuid: "792d15b325a043c68dc06417cf438b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10d37c6bd6745659a9a4a7fd97208d9"
I20260620 15:03:20.276909 16600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "185b1c37dc1f4a0ead36d26976fd54cf" candidate_uuid: "792d15b325a043c68dc06417cf438b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401c08ee81bf452097fc53c65dd88a4f"
I20260620 15:03:20.277091 16600 raft_consensus.cc:3037] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:20.277086 16333 raft_consensus.cc:3037] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:20.279398 16333 raft_consensus.cc:2454] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 792d15b325a043c68dc06417cf438b11 in term 1.
I20260620 15:03:20.279439 16600 raft_consensus.cc:2454] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 792d15b325a043c68dc06417cf438b11 in term 1.
I20260620 15:03:20.279801 16403 leader_election.cc:302] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 792d15b325a043c68dc06417cf438b11, b10d37c6bd6745659a9a4a7fd97208d9; no voters:
I20260620 15:03:20.280108 16671 raft_consensus.cc:2781] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:20.280578 16671 raft_consensus.cc:686] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 1 LEADER]: Becoming Leader. State: Replica: 792d15b325a043c68dc06417cf438b11, State: Running, Role: LEADER
I20260620 15:03:20.281136 16671 consensus_queue.cc:237] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } }
I20260620 15:03:20.285408 16188 catalog_manager.cc:5495] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 792d15b325a043c68dc06417cf438b11 (127.2.70.2). New cstate: current_term: 1 leader_uuid: "792d15b325a043c68dc06417cf438b11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } health_report { overall_health: UNKNOWN } } }
W20260620 15:03:20.317294 16379 tablet.cc:2369] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:03:20.318529 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.312843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.2.70.2:42041)
0620 15:03:20.313082 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.313104 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.313124 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:03:20.313380 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.313384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.313465 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.313655 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:03:20.314060 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.314795 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.316175 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.316579 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.316709 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.317262 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:03:20.317897 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:03:20.318009 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.318163 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.318230 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:03:20.326647 16600 raft_consensus.cc:1261] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Refusing update from remote peer 792d15b325a043c68dc06417cf438b11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:20.326666 16333 raft_consensus.cc:1261] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Refusing update from remote peer 792d15b325a043c68dc06417cf438b11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:20.327592 16675 consensus_queue.cc:1035] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:20.327993 16671 consensus_queue.cc:1035] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:03:20.331586 16513 tablet.cc:2369] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:03:20.339368 16678 mvcc.cc:205] Tried to move back new op lower bound from 7298940110127661056 to 7298940109955805184. Current Snapshot: MvccSnapshot[applied={T|T < 7298940110127661056}]
I20260620 15:03:20.341656 16680 mvcc.cc:205] Tried to move back new op lower bound from 7298940110127661056 to 7298940109955805184. Current Snapshot: MvccSnapshot[applied={T|T < 7298940110127661056}]
I20260620 15:03:20.343542 16679 mvcc.cc:205] Tried to move back new op lower bound from 7298940110127661056 to 7298940109955805184. Current Snapshot: MvccSnapshot[applied={T|T < 7298940110127661056}]
I20260620 15:03:20.354677 16189 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:03:20.354969 16189 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:03:20.358508 16189 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Sending DeleteTablet for 3 replicas of tablet 185b1c37dc1f4a0ead36d26976fd54cf
I20260620 15:03:20.360050 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 16250
I20260620 15:03:20.360211 16313 tablet_service.cc:1514] Processing DeleteTablet for tablet 185b1c37dc1f4a0ead36d26976fd54cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:20 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:58086
I20260620 15:03:20.360961 16580 tablet_service.cc:1514] Processing DeleteTablet for tablet 185b1c37dc1f4a0ead36d26976fd54cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:20 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:49122
I20260620 15:03:20.361243 16447 tablet_service.cc:1514] Processing DeleteTablet for tablet 185b1c37dc1f4a0ead36d26976fd54cf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:20 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:37890
I20260620 15:03:20.371579 16689 tablet_replica.cc:330] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: stopping tablet replica
I20260620 15:03:20.372169 16687 tablet_replica.cc:330] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: stopping tablet replica
I20260620 15:03:20.372529 16690 tablet_replica.cc:330] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: stopping tablet replica
I20260620 15:03:20.372810 16687 raft_consensus.cc:2229] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:03:20.373016 16690 raft_consensus.cc:2229] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:03:20.373278 16689 raft_consensus.cc:2229] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:03:20.373385 16687 raft_consensus.cc:2258] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:20.373906 16690 raft_consensus.cc:2258] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:20.376161 16690 ts_tablet_manager.cc:1905] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:20.376155 16687 ts_tablet_manager.cc:1905] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:20.379019 16689 raft_consensus.cc:2258] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:20.384230 16690 ts_tablet_manager.cc:1918] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:20.384501 16690 log.cc:1196] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/185b1c37dc1f4a0ead36d26976fd54cf
I20260620 15:03:20.384611 16687 ts_tablet_manager.cc:1918] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:20.384845 16687 log.cc:1196] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/185b1c37dc1f4a0ead36d26976fd54cf
I20260620 15:03:20.385295 16690 ts_tablet_manager.cc:1939] T 185b1c37dc1f4a0ead36d26976fd54cf P 792d15b325a043c68dc06417cf438b11: Deleting consensus metadata
I20260620 15:03:20.385428 16687 ts_tablet_manager.cc:1939] T 185b1c37dc1f4a0ead36d26976fd54cf P b10d37c6bd6745659a9a4a7fd97208d9: Deleting consensus metadata
Jun 20 15:03:20 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:03:20.387286 16173 catalog_manager.cc:4841] TS 792d15b325a043c68dc06417cf438b11 (127.2.70.2:42041): tablet 185b1c37dc1f4a0ead36d26976fd54cf (table test-table [id=ffcc81e61f6644e087738eecc7f52c9b]) successfully deleted
I20260620 15:03:20.387566 16689 ts_tablet_manager.cc:1905] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:20.387645 16176 catalog_manager.cc:4841] TS b10d37c6bd6745659a9a4a7fd97208d9 (127.2.70.1:33045): tablet 185b1c37dc1f4a0ead36d26976fd54cf (table test-table [id=ffcc81e61f6644e087738eecc7f52c9b]) successfully deleted
I20260620 15:03:20.394676 16689 ts_tablet_manager.cc:1918] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:20.394843 16689 log.cc:1196] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/185b1c37dc1f4a0ead36d26976fd54cf
I20260620 15:03:20.395116 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.362057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.2.70.1:33045)
0620 15:03:20.362254 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.362264 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.374359 (+ 12095us) server_negotiation.cc:407] Connection header received
0620 15:03:20.374700 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.374708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.374803 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.374882 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:20.376845 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.378048 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.380316 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.380974 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.381100 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.388857 (+ 7757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:20.388868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:20.388887 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:20.388926 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:20.391991 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.392691 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.392707 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.392711 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.392836 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.393235 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.393241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.393244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.393753 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:20.394059 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.394373 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.394601 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 15:03:20.395368 16689 ts_tablet_manager.cc:1939] T 185b1c37dc1f4a0ead36d26976fd54cf P 401c08ee81bf452097fc53c65dd88a4f: Deleting consensus metadata
I20260620 15:03:20.395843 16373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:20.397262 16173 catalog_manager.cc:4841] TS 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859): tablet 185b1c37dc1f4a0ead36d26976fd54cf (table test-table [id=ffcc81e61f6644e087738eecc7f52c9b]) successfully deleted
I20260620 15:03:20.489699 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 16250
I20260620 15:03:20.510166 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 16384
Jun 20 15:03:20 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:03:20.530663 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.511302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37918 (local address 127.2.70.2:42041)
0620 15:03:20.511592 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.511623 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.512301 (+ 678us) server_negotiation.cc:407] Connection header received
0620 15:03:20.512500 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.512505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.512648 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.512788 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:20.514744 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.516016 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.517786 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.518509 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.518663 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.523270 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:20.523298 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:20.523322 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:20.523378 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:20.527369 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.528123 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.528136 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.528141 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.528308 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.528722 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.528728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.528731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.529237 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:20.529823 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.530003 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.530237 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260620 15:03:20.531411 16507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:20.664623 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 16384
I20260620 15:03:20.683266 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 16517
Jun 20 15:03:20 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:03:20.698331 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:20.684036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.2.70.3:34859)
0620 15:03:20.684218 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:20.684222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:20.684256 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:03:20.684425 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:20.684429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:20.684504 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:20.684628 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:20.685982 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.686722 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.688062 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:20.688462 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:20.688554 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:20.692881 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:20.692897 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:20.692914 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:20.692954 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:20.695652 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.696313 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.696321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.696323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.696497 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:20.696833 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:20.696836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:20.696838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:20.697269 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:20.697475 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:20.697902 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 15:03:20.698070 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:03:20.699002 16640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:20.789188 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 16517
I20260620 15:03:20.805917 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 16157
I20260620 15:03:20.807088 16219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:20.917271 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 16157
I20260620 15:03:20.938743 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34541
--webserver_interface=127.2.70.62
--webserver_port=37113
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:21.276463 16695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:21.276757 16695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:21.276846 16695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:21.284381 16695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:21.284502 16695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:21.284551 16695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:21.284616 16695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:21.284653 16695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:21.295243 16695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=37113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:21.296502 16695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:21.297767 16695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:21.320246 16701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:21.320245 16704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:21.320262 16702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:21.321136 16695 server_base.cc:1034] running on GCE node
I20260620 15:03:21.322088 16695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:21.323714 16695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:21.324928 16695 hybrid_clock.cc:648] HybridClock initialized: now 1781967801324842 us; error 66 us; skew 500 ppm
I20260620 15:03:21.325359 16695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:21 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:21.328907 16695 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:03:21.330634 16695 webserver.cc:460] Webserver started at http://127.2.70.62:37113/ using document root <none> and password file <none>
I20260620 15:03:21.331429 16695 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:21.331593 16695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:21.337685 16695 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 15:03:21.340664 16711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:21.342180 16695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:21.342376 16695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "d4210f48d96e43d2b11656eb54a3e6b2"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "8c55fbe778456632dad452ae02c6d3e7"
server_key_iv: "c44757842988c670d54ec077163730ca"
server_key_version: "encryptionkey@0"
I20260620 15:03:21.343516 16695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:21.366852 16695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:21.371302 16695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:21.371611 16695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:21.391140 16695 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:34541
I20260620 15:03:21.391166 16762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:34541 every 8 connection(s)
I20260620 15:03:21.392714 16695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:03:21.399765 16763 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:03:21.402541 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 16695
I20260620 15:03:21.403227 16763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Bootstrap starting.
I20260620 15:03:21.403406 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:33045
--local_ip_for_outbound_sockets=127.2.70.1
--tserver_master_addrs=127.2.70.62:34541
--webserver_port=34827
--webserver_interface=127.2.70.1
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:03:21.423507 16763 log.cc:824] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:21.435930 16763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:03:21.436496 16763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Bootstrap complete.
I20260620 15:03:21.445932 16763 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:21.446791 16763 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4210f48d96e43d2b11656eb54a3e6b2, State: Initialized, Role: FOLLOWER
I20260620 15:03:21.447474 16763 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:21.447733 16763 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:03:21.447860 16763 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:03:21.448045 16763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:03:21.451980 16763 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:21.452514 16763 leader_election.cc:302] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4210f48d96e43d2b11656eb54a3e6b2; no voters:
I20260620 15:03:21.453166 16763 leader_election.cc:288] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:03:21.453382 16767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:03:21.454641 16767 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 2 LEADER]: Becoming Leader. State: Replica: d4210f48d96e43d2b11656eb54a3e6b2, State: Running, Role: LEADER
I20260620 15:03:21.455281 16767 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:21.455554 16763 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:03:21.457868 16768 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } } }
I20260620 15:03:21.458252 16768 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:21.462056 16769 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4210f48d96e43d2b11656eb54a3e6b2. Latest consensus state: current_term: 2 leader_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } } }
I20260620 15:03:21.462387 16769 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:21.463932 16772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:03:21.477574 16772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:03:21.479557 16772 catalog_manager.cc:1259] Loaded cluster ID: 000a03048eaa498b8d82da5a52a11bf2
I20260620 15:03:21.479780 16772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:03:21.484114 16772 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 15:03:21.484293 16784 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:03:21.495446 16772 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 15:03:21.502676 16772 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:03:21.504081 16772 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Loaded TSK: 0
I20260620 15:03:21.505290 16772 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:03:21.778463 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:21.778739 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:21.778888 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:21.786993 16765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:21.787104 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:21.787290 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:03:21.797441 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:33045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=34827
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:21.798799 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:21.800175 16765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:21.813966 16790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:21.813918 16791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:21.814170 16793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:21.815117 16765 server_base.cc:1034] running on GCE node
I20260620 15:03:21.815810 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:21.816912 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:21.818146 16765 hybrid_clock.cc:648] HybridClock initialized: now 1781967801818079 us; error 61 us; skew 500 ppm
I20260620 15:03:21.818496 16765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:21 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:21.828753 16765 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:03:21.830719 16765 webserver.cc:460] Webserver started at http://127.2.70.1:34827/ using document root <none> and password file <none>
I20260620 15:03:21.831710 16765 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:21.831862 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:21.839262 16765 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:03:21.842981 16800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:21.844645 16765 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:03:21.844813 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "b10d37c6bd6745659a9a4a7fd97208d9"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "248316872791e47636323555505dfa52"
server_key_iv: "c79bf2a7b493afd5e6912418f322153d"
server_key_version: "encryptionkey@0"
I20260620 15:03:21.846230 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:21.875507 16765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:21.879707 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:21.879935 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:21.880935 16765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:21.882349 16765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:21.882433 16765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:21.882512 16765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:21.882570 16765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:21.916030 16765 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:33045
I20260620 15:03:21.916141 16912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:33045 every 8 connection(s)
I20260620 15:03:21.917526 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:03:21.928050 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 16765
I20260620 15:03:21.928817 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:42041
--local_ip_for_outbound_sockets=127.2.70.2
--tserver_master_addrs=127.2.70.62:34541
--webserver_port=37347
--webserver_interface=127.2.70.2
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:03:21 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967801, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:21.943384 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:21.920733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:35001 (local address 127.2.70.62:34541)
0620 15:03:21.921283 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:21.921295 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:21.922110 (+ 815us) server_negotiation.cc:407] Connection header received
0620 15:03:21.924199 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:21.924213 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:21.924544 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:21.924842 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:21.926493 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:21.927726 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:21.930210 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:21.930833 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:21.930967 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:21.935217 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:21.935231 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:21.935255 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:21.935318 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:21.938740 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:21.939750 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:21.939757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:21.939763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:21.939916 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:21.940394 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:21.940397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:21.940399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:21.940849 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:21.940998 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:21.942036 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0620 15:03:21.942362 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260620 15:03:21.943905 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:21.921021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:35001)
0620 15:03:21.921933 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:21.921978 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:21.923695 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:21.924976 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:21.924984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:21.925694 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:21.926246 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:21.926264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:21.927909 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:21.927915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:21.930062 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:21.930078 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:21.931080 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:21.931090 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:21.931410 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:21.932173 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:21.932209 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:21.934900 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:21.938898 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:21.938910 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:21.938929 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:21.939609 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:21.940029 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:21.940036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:21.940042 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:21.940280 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:21.941020 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:21.941031 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:21.941531 (+ 500us) client_negotiation.cc:769] Sending connection context
0620 15:03:21.941843 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 15:03:21.942423 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":392,"threads_started":1}
I20260620 15:03:21.946772 16913 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34541
I20260620 15:03:21.947083 16913 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:21.947839 16913 heartbeater.cc:511] Master 127.2.70.62:34541 requested a full tablet report, sending...
I20260620 15:03:21.950071 16728 ts_manager.cc:194] Registered new tserver with Master: b10d37c6bd6745659a9a4a7fd97208d9 (127.2.70.1:33045)
I20260620 15:03:21.951634 16728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:35001
W20260620 15:03:22.264976 16917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:22.265300 16917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:22.265477 16917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:22.272691 16917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:22.272794 16917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:22.272953 16917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:03:22.282718 16917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:42041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=37347
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:22.284000 16917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:22.285367 16917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:22.298264 16925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:22.298341 16922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:22.298300 16923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:22.299161 16917 server_base.cc:1034] running on GCE node
I20260620 15:03:22.299741 16917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:22.300606 16917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:22.301832 16917 hybrid_clock.cc:648] HybridClock initialized: now 1781967802301761 us; error 57 us; skew 500 ppm
I20260620 15:03:22.302222 16917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:22 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:22.305341 16917 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:03:22.306813 16917 webserver.cc:460] Webserver started at http://127.2.70.2:37347/ using document root <none> and password file <none>
I20260620 15:03:22.307504 16917 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:22.307617 16917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:22.313205 16917 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 15:03:22.316075 16932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:22.317559 16917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 15:03:22.317719 16917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "792d15b325a043c68dc06417cf438b11"
format_stamp: "Formatted at 2026-06-20 15:03:19 on dist-test-slave-n5vv"
server_key: "c833e35d9535b10c7bec484b99cb1634"
server_key_iv: "3029a71d462228c6eb02c096765963c4"
server_key_version: "encryptionkey@0"
I20260620 15:03:22.318676 16917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:22.334669 16917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:22.338881 16917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:22.339104 16917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:22.340011 16917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:22.341248 16917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:22.341332 16917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:22.341421 16917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:22.341499 16917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:22.374954 16917 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:42041
I20260620 15:03:22.375032 17044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:42041 every 8 connection(s)
I20260620 15:03:22.376684 16917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:03:22.386785 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 16917
I20260620 15:03:22.387542 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:34859
--local_ip_for_outbound_sockets=127.2.70.3
--tserver_master_addrs=127.2.70.62:34541
--webserver_port=40015
--webserver_interface=127.2.70.3
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:03:22 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:22.398234 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.379436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:36877 (local address 127.2.70.62:34541)
0620 15:03:22.379653 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:22.379660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:22.380847 (+ 1187us) server_negotiation.cc:407] Connection header received
0620 15:03:22.382708 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:22.382712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:22.382777 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:22.382865 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:22.384304 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.384983 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.386630 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.387291 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.387444 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.391137 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:22.391147 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:22.391155 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:22.391204 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:22.395043 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:22.395838 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:22.395844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:22.395846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:22.395969 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:22.396431 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:22.396440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:22.396442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:22.396735 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:22.396898 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:22.397644 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 15:03:22.397879 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:03:22.399305 17047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.379980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:36877)
0620 15:03:22.380690 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:22.380726 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:22.382460 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:22.383083 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:22.383091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:22.383600 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:22.384159 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.384176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.385153 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.385158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.386481 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.386492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.387488 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.387492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.387682 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.388477 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:22.388499 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:22.390885 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:22.395220 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:22.395230 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:22.395253 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:22.395695 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:22.396087 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:22.396094 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:22.396098 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:22.396301 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:22.396869 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:22.396878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:22.397223 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 15:03:22.397491 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 15:03:22.398071 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":277,"threads_started":1}
I20260620 15:03:22.400873 17045 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34541
I20260620 15:03:22.401206 17045 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:22.402002 17045 heartbeater.cc:511] Master 127.2.70.62:34541 requested a full tablet report, sending...
I20260620 15:03:22.403393 16728 ts_manager.cc:194] Registered new tserver with Master: 792d15b325a043c68dc06417cf438b11 (127.2.70.2:42041)
I20260620 15:03:22.404269 16728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:36877
W20260620 15:03:22.727682 17048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:22.727982 17048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:22.728092 17048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:22.735913 17048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:22.736032 17048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:22.736145 17048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:03:22.746351 17048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:34859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=40015
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:22.747805 17048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:22.749119 17048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:22.762737 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:22.762777 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:22.762985 17053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:22.763687 17048 server_base.cc:1034] running on GCE node
I20260620 15:03:22.764353 17048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:22.765383 17048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:22.766619 17048 hybrid_clock.cc:648] HybridClock initialized: now 1781967802766551 us; error 58 us; skew 500 ppm
I20260620 15:03:22.766937 17048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:22 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:22.769956 17048 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:03:22.771725 17048 webserver.cc:460] Webserver started at http://127.2.70.3:40015/ using document root <none> and password file <none>
I20260620 15:03:22.772488 17048 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:22.772655 17048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:22.777959 17048 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:22.780546 17063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:22.781989 17048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:22.782181 17048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "401c08ee81bf452097fc53c65dd88a4f"
format_stamp: "Formatted at 2026-06-20 15:03:20 on dist-test-slave-n5vv"
server_key: "bbbb344b2895750e2783ab940ba9eb32"
server_key_iv: "590755bc09a6271a5a7e58e18fc10523"
server_key_version: "encryptionkey@0"
I20260620 15:03:22.783156 17048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:22.820994 17048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:22.825332 17048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:22.825670 17048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:22.826722 17048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:22.828080 17048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:22.828217 17048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:22.828341 17048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:22.828428 17048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:22.861729 17048 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:34859
I20260620 15:03:22.861814 17175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:34859 every 8 connection(s)
I20260620 15:03:22.863210 17048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:03:22.869788 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 17048
Jun 20 15:03:22 dist-test-slave-n5vv krb5kdc[16144](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967802, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:22.884534 17180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.870834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.2.70.62:34541)
0620 15:03:22.871665 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:22.871673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:22.871693 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:03:22.871771 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:22.871778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:22.871846 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:22.871977 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:22.873446 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.874228 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.875580 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.876003 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.876142 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.878839 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:22.878844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:22.878847 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:22.878883 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:22.881849 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:22.882478 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:22.882491 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:22.882494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:22.882616 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:22.882975 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:22.882981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:22.882984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:22.883282 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:22.883535 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:22.883866 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 15:03:22.884100 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":699,"spinlock_wait_cycles":2816,"thread_start_us":569,"threads_started":1}
I20260620 15:03:22.884824 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.865929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:46195 (local address 127.2.70.62:34541)
0620 15:03:22.866159 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:22.866165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:22.867491 (+ 1326us) server_negotiation.cc:407] Connection header received
0620 15:03:22.869273 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:22.869278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:22.869351 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:22.869441 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:22.871799 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.872590 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.874225 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.874634 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.874761 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.878325 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:22.878330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:22.878333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:22.878362 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:22.881748 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:22.882483 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:22.882493 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:22.882496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:22.882646 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:22.883097 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:22.883101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:22.883103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:22.883371 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:22.883518 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:22.884231 (+ 713us) server_negotiation.cc:299] Negotiation successful
0620 15:03:22.884433 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":61,"spinlock_wait_cycles":14336}
I20260620 15:03:22.885946 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.866579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:46195)
0620 15:03:22.867321 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:22.867358 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:22.868993 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:22.869778 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:22.869790 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:22.870547 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:22.871457 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.871483 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.872753 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.872758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.874070 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.874088 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.874864 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.874871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.875102 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.875783 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:22.875805 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:22.878101 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:22.881898 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:22.881907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:22.881924 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:22.882351 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:22.882759 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:22.882764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:22.882768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:22.882962 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:22.883500 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:22.883508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:22.883853 (+ 345us) client_negotiation.cc:769] Sending connection context
0620 15:03:22.884111 (+ 258us) client_negotiation.cc:240] Negotiation successful
0620 15:03:22.884659 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":281,"threads_started":1}
I20260620 15:03:22.887382 17176 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34541
I20260620 15:03:22.887723 17176 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:22.888489 17176 heartbeater.cc:511] Master 127.2.70.62:34541 requested a full tablet report, sending...
I20260620 15:03:22.889825 16728 ts_manager.cc:194] Registered new tserver with Master: 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859)
I20260620 15:03:22.890641 16728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:46195
I20260620 15:03:22.899580 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:03:22.906484 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.900562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.2.70.62:34541)
0620 15:03:22.900801 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:22.900809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:22.901119 (+ 310us) server_negotiation.cc:407] Connection header received
0620 15:03:22.901348 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:22.901351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:22.901536 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:22.901696 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:03:22.902128 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.902846 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.904317 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.904730 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.904867 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.905365 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:03:22.905930 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:03:22.906105 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:22.906158 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 15:03:22.906238 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 15:03:22.907375 16728 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:03:22.915117 16728 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:03:22.940292 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.927002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34859 (local address 127.0.0.1:43694)
0620 15:03:22.929091 (+ 2089us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:22.929125 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:22.929338 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:22.930414 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:22.930425 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:22.930466 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:22.930951 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.930967 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.933669 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.933678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.936951 (+ 3273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.936972 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.939287 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.939293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.939496 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.939560 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 15:03:22.939702 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 15:03:22.939909 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1844,"thread_start_us":1714,"threads_started":1}
I20260620 15:03:22.941079 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.927367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43694 (local address 127.2.70.3:34859)
0620 15:03:22.928239 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:22.928245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:22.929252 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 15:03:22.929569 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:22.929584 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:22.929863 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:22.930041 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:22.931212 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.933383 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.937160 (+ 3777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.938729 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.938858 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.939096 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:22.940545 (+ 1449us) server_negotiation.cc:299] Negotiation successful
0620 15:03:22.940742 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":541,"threads_started":1}
I20260620 15:03:22.942711 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.928781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33045 (local address 127.0.0.1:48040)
0620 15:03:22.933738 (+ 4957us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:22.933755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:22.933907 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:22.934678 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:22.934683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:22.934710 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:22.935134 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.935151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.937264 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.937269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.939391 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.939407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.941972 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.941976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.942102 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.942124 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:03:22.942237 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 15:03:22.942366 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4824,"thread_start_us":4663,"threads_started":1}
I20260620 15:03:22.942772 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.928044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.2.70.1:33045)
0620 15:03:22.928709 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:22.928717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:22.933868 (+ 5151us) server_negotiation.cc:407] Connection header received
0620 15:03:22.934091 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:22.934101 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:22.934445 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:22.934670 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:22.935993 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.937013 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.939721 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.941234 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.941336 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.941589 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:22.942280 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 15:03:22.942439 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":369,"threads_started":1}
I20260620 15:03:22.946238 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.929768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:42041 (local address 127.0.0.1:55776)
0620 15:03:22.937683 (+ 7915us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:22.937712 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:22.937850 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:22.938838 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:22.938843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:22.938871 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:22.939364 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.939380 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.941099 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.941105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.943314 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:22.943328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.945418 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.945423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:22.945623 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.945647 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:03:22.945756 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:03:22.945893 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7729,"thread_start_us":9729,"threads_started":1}
I20260620 15:03:22.946269 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:22.929956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.2.70.2:42041)
0620 15:03:22.938344 (+ 8388us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:22.938352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:22.938380 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:03:22.938462 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:22.938467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:22.938653 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:22.938851 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:22.939584 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.940966 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.943494 (+ 2528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:22.945010 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:22.945137 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:22.945347 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:22.945801 (+ 454us) server_negotiation.cc:299] Negotiation successful
0620 15:03:22.945974 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8243,"thread_start_us":8149,"threads_started":1}
I20260620 15:03:22.945803 17111 tablet_service.cc:1467] Processing CreateTablet for tablet 3921962614bb4719ab5b4a6e18556fc3 (DEFAULT_TABLE table=test-table [id=151156c2933344ca83dd7c5f664b5638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:22.946612 16848 tablet_service.cc:1467] Processing CreateTablet for tablet 3921962614bb4719ab5b4a6e18556fc3 (DEFAULT_TABLE table=test-table [id=151156c2933344ca83dd7c5f664b5638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:22.947379 17111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3921962614bb4719ab5b4a6e18556fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:22.948042 16848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3921962614bb4719ab5b4a6e18556fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:22.949782 16980 tablet_service.cc:1467] Processing CreateTablet for tablet 3921962614bb4719ab5b4a6e18556fc3 (DEFAULT_TABLE table=test-table [id=151156c2933344ca83dd7c5f664b5638]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:22.951045 16980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3921962614bb4719ab5b4a6e18556fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:22.954846 16913 heartbeater.cc:503] Master 127.2.70.62:34541 was elected leader, sending a full tablet report...
I20260620 15:03:22.965541 17192 tablet_bootstrap.cc:492] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Bootstrap starting.
I20260620 15:03:22.969985 17193 tablet_bootstrap.cc:492] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Bootstrap starting.
I20260620 15:03:22.970180 17194 tablet_bootstrap.cc:492] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Bootstrap starting.
I20260620 15:03:22.971320 17192 tablet_bootstrap.cc:654] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:22.972652 17192 log.cc:824] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:22.973531 17194 tablet_bootstrap.cc:654] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:22.973978 17193 tablet_bootstrap.cc:654] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:22.974802 17194 log.cc:824] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:22.975212 17193 log.cc:824] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:22.975641 17192 tablet_bootstrap.cc:492] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: No bootstrap required, opened a new log
I20260620 15:03:22.975929 17192 ts_tablet_manager.cc:1397] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260620 15:03:22.978199 17194 tablet_bootstrap.cc:492] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: No bootstrap required, opened a new log
I20260620 15:03:22.978470 17194 ts_tablet_manager.cc:1397] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:03:22.978832 17193 tablet_bootstrap.cc:492] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: No bootstrap required, opened a new log
I20260620 15:03:22.979091 17193 ts_tablet_manager.cc:1397] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 15:03:22.984115 17192 raft_consensus.cc:348] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:22.984496 17192 raft_consensus.cc:374] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:22.984612 17192 raft_consensus.cc:729] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b10d37c6bd6745659a9a4a7fd97208d9, State: Initialized, Role: FOLLOWER
I20260620 15:03:22.985265 17192 consensus_queue.cc:260] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:22.987154 17192 ts_tablet_manager.cc:1428] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260620 15:03:22.989077 17194 raft_consensus.cc:348] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:22.989528 17194 raft_consensus.cc:374] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:22.989639 17194 raft_consensus.cc:729] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 792d15b325a043c68dc06417cf438b11, State: Initialized, Role: FOLLOWER
I20260620 15:03:22.990087 17193 raft_consensus.cc:348] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:22.990513 17193 raft_consensus.cc:374] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:22.990350 17194 consensus_queue.cc:260] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:22.990612 17193 raft_consensus.cc:729] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401c08ee81bf452097fc53c65dd88a4f, State: Initialized, Role: FOLLOWER
I20260620 15:03:22.991750 17193 consensus_queue.cc:260] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:22.992902 17176 heartbeater.cc:503] Master 127.2.70.62:34541 was elected leader, sending a full tablet report...
I20260620 15:03:22.993914 17193 ts_tablet_manager.cc:1428] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260620 15:03:22.997399 17045 heartbeater.cc:503] Master 127.2.70.62:34541 was elected leader, sending a full tablet report...
I20260620 15:03:22.998175 17194 ts_tablet_manager.cc:1428] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Time spent starting tablet: real 0.020s user 0.006s sys 0.013s
W20260620 15:03:23.114903 17177 tablet.cc:2369] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:03:23.128823 17046 tablet.cc:2369] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:03:23.169839 16914 tablet.cc:2369] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:03:23.277354 17199 raft_consensus.cc:482] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:03:23.277756 17199 raft_consensus.cc:504] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:23.279547 17199 leader_election.cc:288] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859), b10d37c6bd6745659a9a4a7fd97208d9 (127.2.70.1:33045)
I20260620 15:03:23.286818 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.279908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:49935 (local address 127.2.70.3:34859)
0620 15:03:23.280149 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:23.280155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:23.280952 (+ 797us) server_negotiation.cc:407] Connection header received
0620 15:03:23.281150 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:23.281158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:23.281249 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:23.281342 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:23.282465 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.283178 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.284909 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.285797 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.285922 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.286044 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:23.286416 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 15:03:23.286538 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:03:23.286814 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.280342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:49095 (local address 127.2.70.1:33045)
0620 15:03:23.280564 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:23.280569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:23.281227 (+ 658us) server_negotiation.cc:407] Connection header received
0620 15:03:23.281383 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:23.281387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:23.281446 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:23.281582 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:23.282237 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.283046 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.284943 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.285804 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.285960 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.286105 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:23.286364 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 15:03:23.286497 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:03:23.287268 17202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.280286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33045 (local address 127.2.70.2:49095)
0620 15:03:23.281107 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:23.281124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:23.281255 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:23.281617 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:23.281621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:23.281641 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:23.282065 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:23.282074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.283198 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.283204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:23.284759 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:23.284771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.286060 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.286069 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:23.286209 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.286233 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:03:23.286310 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 15:03:23.286459 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":5,"thread_start_us":431,"threads_started":1}
I20260620 15:03:23.287349 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.279948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34859 (local address 127.2.70.2:49935)
0620 15:03:23.280794 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:23.280818 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:23.280995 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:23.281744 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:23.281748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:23.281815 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:23.282309 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:23.282321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.283321 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.283324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:23.284751 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:23.284762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.286023 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.286031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:23.286233 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.286257 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:03:23.286365 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 15:03:23.286489 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":43,"thread_start_us":378,"threads_started":1}
I20260620 15:03:23.288434 17131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3921962614bb4719ab5b4a6e18556fc3" candidate_uuid: "792d15b325a043c68dc06417cf438b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401c08ee81bf452097fc53c65dd88a4f" is_pre_election: true
I20260620 15:03:23.288432 16868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3921962614bb4719ab5b4a6e18556fc3" candidate_uuid: "792d15b325a043c68dc06417cf438b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" is_pre_election: true
I20260620 15:03:23.289019 17131 raft_consensus.cc:2454] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 792d15b325a043c68dc06417cf438b11 in term 0.
I20260620 15:03:23.289044 16868 raft_consensus.cc:2454] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 792d15b325a043c68dc06417cf438b11 in term 0.
I20260620 15:03:23.289306 17198 raft_consensus.cc:482] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:03:23.289558 17198 raft_consensus.cc:504] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:23.289834 16933 leader_election.cc:302] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401c08ee81bf452097fc53c65dd88a4f, 792d15b325a043c68dc06417cf438b11; no voters:
I20260620 15:03:23.290243 17199 raft_consensus.cc:2781] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:03:23.290422 17199 raft_consensus.cc:482] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:03:23.290567 17199 raft_consensus.cc:3037] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:23.291318 17198 leader_election.cc:288] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859), 792d15b325a043c68dc06417cf438b11 (127.2.70.2:42041)
I20260620 15:03:23.294409 17199 raft_consensus.cc:504] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:23.295346 17199 leader_election.cc:288] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [CANDIDATE]: Term 1 election: Requested vote from peers 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859), b10d37c6bd6745659a9a4a7fd97208d9 (127.2.70.1:33045)
I20260620 15:03:23.295723 17131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3921962614bb4719ab5b4a6e18556fc3" candidate_uuid: "792d15b325a043c68dc06417cf438b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401c08ee81bf452097fc53c65dd88a4f"
I20260620 15:03:23.295853 16868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3921962614bb4719ab5b4a6e18556fc3" candidate_uuid: "792d15b325a043c68dc06417cf438b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b10d37c6bd6745659a9a4a7fd97208d9"
I20260620 15:03:23.295954 17131 raft_consensus.cc:3037] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:23.296082 16868 raft_consensus.cc:3037] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:23.298841 16868 raft_consensus.cc:2454] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 792d15b325a043c68dc06417cf438b11 in term 1.
I20260620 15:03:23.299109 17131 raft_consensus.cc:2454] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 792d15b325a043c68dc06417cf438b11 in term 1.
I20260620 15:03:23.299374 16936 leader_election.cc:302] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 792d15b325a043c68dc06417cf438b11, b10d37c6bd6745659a9a4a7fd97208d9; no voters:
I20260620 15:03:23.299988 17199 raft_consensus.cc:2781] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:23.304103 17203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.291594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34859 (local address 127.2.70.1:60869)
0620 15:03:23.296035 (+ 4441us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:23.296058 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:23.296211 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:23.296598 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:23.296605 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:23.296646 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:23.297084 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:23.297095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.298649 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.298657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:23.300972 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:23.300987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.303135 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.303142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:23.303388 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.303434 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:03:23.303550 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 15:03:23.303694 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4304,"mutex_wait_us":44,"thread_start_us":7463,"threads_started":1}
I20260620 15:03:23.304150 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.291667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:60869 (local address 127.2.70.3:34859)
0620 15:03:23.291823 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:23.291827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:23.296159 (+ 4332us) server_negotiation.cc:407] Connection header received
0620 15:03:23.296354 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:23.296363 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:23.296448 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:23.296562 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:23.297232 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.298495 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.301152 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.302900 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.303065 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.303213 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:23.303585 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 15:03:23.303779 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:03:23.304662 17131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3921962614bb4719ab5b4a6e18556fc3" candidate_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401c08ee81bf452097fc53c65dd88a4f" is_pre_election: true
I20260620 15:03:23.304981 17131 raft_consensus.cc:2379] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b10d37c6bd6745659a9a4a7fd97208d9 in current term 1: Already voted for candidate 792d15b325a043c68dc06417cf438b11 in this term.
I20260620 15:03:23.305783 17199 raft_consensus.cc:686] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 1 LEADER]: Becoming Leader. State: Replica: 792d15b325a043c68dc06417cf438b11, State: Running, Role: LEADER
I20260620 15:03:23.306381 17199 consensus_queue.cc:237] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } }
I20260620 15:03:23.307376 17204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.299678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:42041 (local address 127.2.70.1:45893)
0620 15:03:23.300607 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:23.300637 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:23.300804 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:23.301283 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:23.301286 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:23.301306 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:23.301676 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:23.301685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.302704 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.302712 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:23.304740 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:23.304753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.306705 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.306709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:23.306845 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.306873 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 15:03:23.306963 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:03:23.307056 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":1,"thread_start_us":497,"threads_started":1}
I20260620 15:03:23.307380 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.299809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:45893 (local address 127.2.70.2:42041)
0620 15:03:23.300025 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:23.300032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:23.300752 (+ 720us) server_negotiation.cc:407] Connection header received
0620 15:03:23.300972 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:23.300977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:23.301065 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:23.301146 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:23.301826 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.302569 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.304906 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.306503 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.306667 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.306756 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:23.307002 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 15:03:23.307131 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:03:23.308269 17000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3921962614bb4719ab5b4a6e18556fc3" candidate_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "792d15b325a043c68dc06417cf438b11" is_pre_election: true
I20260620 15:03:23.310441 16801 leader_election.cc:302] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b10d37c6bd6745659a9a4a7fd97208d9; no voters: 401c08ee81bf452097fc53c65dd88a4f, 792d15b325a043c68dc06417cf438b11
I20260620 15:03:23.310839 17198 raft_consensus.cc:2726] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 15:03:23.311544 16728 catalog_manager.cc:5495] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 792d15b325a043c68dc06417cf438b11 (127.2.70.2). New cstate: current_term: 1 leader_uuid: "792d15b325a043c68dc06417cf438b11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "792d15b325a043c68dc06417cf438b11" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 } health_report { overall_health: UNKNOWN } } }
I20260620 15:03:23.394253 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.388335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.2.70.2:42041)
0620 15:03:23.388507 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:23.388511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:23.388555 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:03:23.388772 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:23.388776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:23.388861 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:23.388979 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:03:23.389345 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.390169 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.391636 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.392279 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.392404 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.392904 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:03:23.393627 (+ 723us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:03:23.393783 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:23.393836 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 15:03:23.393938 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:03:23.404218 17131 raft_consensus.cc:1261] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Refusing update from remote peer 792d15b325a043c68dc06417cf438b11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:23.404243 16868 raft_consensus.cc:1261] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Refusing update from remote peer 792d15b325a043c68dc06417cf438b11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:23.405273 17205 consensus_queue.cc:1035] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401c08ee81bf452097fc53c65dd88a4f" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:23.405695 17199 consensus_queue.cc:1035] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b10d37c6bd6745659a9a4a7fd97208d9" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:23.418802 17207 mvcc.cc:205] Tried to move back new op lower bound from 7298940122732732416 to 7298940122348208128. Current Snapshot: MvccSnapshot[applied={T|T < 7298940122732732416}]
I20260620 15:03:23.435714 16727 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:03:23.435985 16727 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:03:23.439858 16727 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Sending DeleteTablet for 3 replicas of tablet 3921962614bb4719ab5b4a6e18556fc3
I20260620 15:03:23.441298 16848 tablet_service.cc:1514] Processing DeleteTablet for tablet 3921962614bb4719ab5b4a6e18556fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:23 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:48040
I20260620 15:03:23.441339 16980 tablet_service.cc:1514] Processing DeleteTablet for tablet 3921962614bb4719ab5b4a6e18556fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:23 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:55776
I20260620 15:03:23.441334 17111 tablet_service.cc:1514] Processing DeleteTablet for tablet 3921962614bb4719ab5b4a6e18556fc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:23 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:43694
I20260620 15:03:23.444010 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 16765
I20260620 15:03:23.454145 17219 tablet_replica.cc:330] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: stopping tablet replica
I20260620 15:03:23.454715 17219 raft_consensus.cc:2229] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:03:23.457417 17220 tablet_replica.cc:330] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: stopping tablet replica
I20260620 15:03:23.457945 17220 raft_consensus.cc:2229] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:03:23.458488 17220 raft_consensus.cc:2258] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:23.459101 17219 raft_consensus.cc:2258] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:23.460804 17218 tablet_replica.cc:330] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: stopping tablet replica
I20260620 15:03:23.462337 17219 ts_tablet_manager.cc:1905] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:23.462707 17220 ts_tablet_manager.cc:1905] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:23.463470 17218 raft_consensus.cc:2229] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:03:23.464095 17218 raft_consensus.cc:2258] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:23.468233 17218 ts_tablet_manager.cc:1905] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:23.470526 17219 ts_tablet_manager.cc:1918] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:23.470793 17219 log.cc:1196] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/3921962614bb4719ab5b4a6e18556fc3
I20260620 15:03:23.471531 17219 ts_tablet_manager.cc:1939] T 3921962614bb4719ab5b4a6e18556fc3 P 792d15b325a043c68dc06417cf438b11: Deleting consensus metadata
I20260620 15:03:23.471541 17220 ts_tablet_manager.cc:1918] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:23.471719 17220 log.cc:1196] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/3921962614bb4719ab5b4a6e18556fc3
I20260620 15:03:23.472298 17220 ts_tablet_manager.cc:1939] T 3921962614bb4719ab5b4a6e18556fc3 P b10d37c6bd6745659a9a4a7fd97208d9: Deleting consensus metadata
I20260620 15:03:23.473824 16712 catalog_manager.cc:4841] TS 792d15b325a043c68dc06417cf438b11 (127.2.70.2:42041): tablet 3921962614bb4719ab5b4a6e18556fc3 (table test-table [id=151156c2933344ca83dd7c5f664b5638]) successfully deleted
I20260620 15:03:23.474357 16715 catalog_manager.cc:4841] TS b10d37c6bd6745659a9a4a7fd97208d9 (127.2.70.1:33045): tablet 3921962614bb4719ab5b4a6e18556fc3 (table test-table [id=151156c2933344ca83dd7c5f664b5638]) successfully deleted
I20260620 15:03:23.480412 17218 ts_tablet_manager.cc:1918] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:23.480623 17218 log.cc:1196] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/3921962614bb4719ab5b4a6e18556fc3
I20260620 15:03:23.481074 17218 ts_tablet_manager.cc:1939] T 3921962614bb4719ab5b4a6e18556fc3 P 401c08ee81bf452097fc53c65dd88a4f: Deleting consensus metadata
I20260620 15:03:23.482828 17188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.444885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.2.70.1:33045)
0620 15:03:23.445081 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:23.445087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:23.460087 (+ 15000us) server_negotiation.cc:407] Connection header received
0620 15:03:23.460277 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:23.460283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:23.460387 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:23.460495 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:23.463061 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.464228 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.468510 (+ 4282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.469303 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.469414 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.474728 (+ 5314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:23.474742 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:23.474762 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:23.474806 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:23.478812 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:23.479786 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:23.479795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:23.479799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:23.479950 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:23.480441 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:23.480447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:23.480450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:23.480974 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:23.481277 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:23.482023 (+ 746us) server_negotiation.cc:299] Negotiation successful
0620 15:03:23.482261 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:03:23.482950 16712 catalog_manager.cc:4841] TS 401c08ee81bf452097fc53c65dd88a4f (127.2.70.3:34859): tablet 3921962614bb4719ab5b4a6e18556fc3 (table test-table [id=151156c2933344ca83dd7c5f664b5638]) successfully deleted
I20260620 15:03:23.483445 16908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:23.614820 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 16765
I20260620 15:03:23.637763 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 16917
I20260620 15:03:23.654515 17191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.638570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.2.70.2:42041)
0620 15:03:23.638759 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:23.638763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:23.638835 (+ 72us) server_negotiation.cc:407] Connection header received
0620 15:03:23.639026 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:23.639034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:23.639125 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:23.639212 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:23.640957 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.641945 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.643825 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.644356 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.644490 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.647972 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:23.647986 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:23.648003 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:23.648088 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:23.651448 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:23.652342 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:23.652350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:23.652352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:23.652541 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:23.652961 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:23.652964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:23.652966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:23.653388 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:23.653683 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:23.653985 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 15:03:23.654197 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:03:23.655268 17040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:23.771875 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 16917
I20260620 15:03:23.790927 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 17048
I20260620 15:03:23.804440 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:23.791715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.2.70.3:34859)
0620 15:03:23.791926 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:23.791931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:23.791951 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:03:23.792192 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:23.792196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:23.792267 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:23.792369 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:23.793765 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.794478 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.795897 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:23.796321 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:23.796455 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:23.799161 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:23.799174 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:23.799190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:23.799223 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:23.801811 (+ 2588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:23.802471 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:23.802479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:23.802481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:23.802606 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:23.802975 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:23.802981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:23.802984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:23.803309 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:23.803681 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:23.803879 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 15:03:23.804107 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:03:23.805112 17171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:23.896104 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 17048
I20260620 15:03:23.912786 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 16695
I20260620 15:03:23.914063 16758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:24.028218 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 16695
I20260620 15:03:24.051628 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34541
--webserver_interface=127.2.70.62
--webserver_port=37113
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:24.385721 17226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:24.386046 17226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:24.386142 17226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:24.393512 17226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:24.393626 17226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:24.393673 17226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:24.393733 17226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:24.393775 17226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:24.403630 17226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=37113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:24.404906 17226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:24.406174 17226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:24.417914 17231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:24.417873 17232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:24.417938 17234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:24.419028 17226 server_base.cc:1034] running on GCE node
I20260620 15:03:24.419806 17226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:24.421301 17226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:24.422593 17226 hybrid_clock.cc:648] HybridClock initialized: now 1781967804422540 us; error 47 us; skew 500 ppm
I20260620 15:03:24.422948 17226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:24 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:24.426044 17226 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:03:24.427541 17226 webserver.cc:460] Webserver started at http://127.2.70.62:37113/ using document root <none> and password file <none>
I20260620 15:03:24.428182 17226 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:24.428288 17226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:24.433837 17226 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 15:03:24.436559 17241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:24.437958 17226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:24.438093 17226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "d4210f48d96e43d2b11656eb54a3e6b2"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "8c55fbe778456632dad452ae02c6d3e7"
server_key_iv: "c44757842988c670d54ec077163730ca"
server_key_version: "encryptionkey@0"
I20260620 15:03:24.439060 17226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:24.470255 17226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:24.474390 17226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:24.474633 17226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:24.493813 17226 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:34541
I20260620 15:03:24.493860 17292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:34541 every 8 connection(s)
I20260620 15:03:24.495322 17226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:03:24.502255 17293 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:03:24.504664 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 17226
I20260620 15:03:24.505628 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:33045
--local_ip_for_outbound_sockets=127.2.70.1
--tserver_master_addrs=127.2.70.62:34541
--webserver_port=34827
--webserver_interface=127.2.70.1
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:03:24.505887 17293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Bootstrap starting.
I20260620 15:03:24.536695 17293 log.cc:824] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:24.557554 17293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:03:24.558205 17293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2: Bootstrap complete.
I20260620 15:03:24.571367 17293 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:24.572314 17293 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4210f48d96e43d2b11656eb54a3e6b2, State: Initialized, Role: FOLLOWER
I20260620 15:03:24.573012 17293 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:24.573271 17293 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:03:24.573397 17293 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:03:24.573611 17293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:03:24.577574 17293 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:24.578118 17293 leader_election.cc:302] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4210f48d96e43d2b11656eb54a3e6b2; no voters:
I20260620 15:03:24.578799 17293 leader_election.cc:288] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:03:24.578943 17297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:03:24.580009 17297 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [term 3 LEADER]: Becoming Leader. State: Replica: d4210f48d96e43d2b11656eb54a3e6b2, State: Running, Role: LEADER
I20260620 15:03:24.580554 17297 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } }
I20260620 15:03:24.581008 17293 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:03:24.583300 17298 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } } }
I20260620 15:03:24.583355 17299 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4210f48d96e43d2b11656eb54a3e6b2. Latest consensus state: current_term: 3 leader_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4210f48d96e43d2b11656eb54a3e6b2" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34541 } } }
I20260620 15:03:24.583712 17298 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:24.583745 17299 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4210f48d96e43d2b11656eb54a3e6b2 [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:24.595044 17304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:03:24.606329 17304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:03:24.608495 17304 catalog_manager.cc:1259] Loaded cluster ID: 000a03048eaa498b8d82da5a52a11bf2
I20260620 15:03:24.608644 17304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 15:03:24.611295 17304 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f252f95f9ad google::LogMessage::Fail() at ??:0
@ 0x7f252f96288a google::LogMessage::SendToLog() at ??:0
@ 0x7f252f95f4de google::LogMessage::Flush() at ??:0
@ 0x7f252f960a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20260620 15:03:24.622258 17315 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f2545392512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f254538f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2545417610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2545417439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f25325ab345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f25325ae91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f25325ae799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2532589eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f25358636db start_thread at ??:0
@ 0x7f252e7f961f clone at ??:0
W20260620 15:03:24.896279 17295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:24.896607 17295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:24.896771 17295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:24.904315 17295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:24.904455 17295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:24.904652 17295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:03:24.914796 17295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:33045
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=34827
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:24.916160 17295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:24.917580 17295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:24.930964 17334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:24.931015 17337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:24.931085 17335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:24.931850 17295 server_base.cc:1034] running on GCE node
I20260620 15:03:24.932479 17295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:24.933431 17295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:24.934684 17295 hybrid_clock.cc:648] HybridClock initialized: now 1781967804934644 us; error 32 us; skew 500 ppm
I20260620 15:03:24.935016 17295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:24 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967804, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:24.938053 17295 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:03:24.939443 17295 webserver.cc:460] Webserver started at http://127.2.70.1:34827/ using document root <none> and password file <none>
I20260620 15:03:24.940157 17295 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:24.940296 17295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:24.945884 17295 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260620 15:03:24.948553 17344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:24.949970 17295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:24.950161 17295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "b10d37c6bd6745659a9a4a7fd97208d9"
format_stamp: "Formatted at 2026-06-20 15:03:18 on dist-test-slave-n5vv"
server_key: "248316872791e47636323555505dfa52"
server_key_iv: "c79bf2a7b493afd5e6912418f322153d"
server_key_version: "encryptionkey@0"
I20260620 15:03:24.951203 17295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:24.971789 17295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:24.975971 17295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:24.976279 17295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:24.977372 17295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:24.978863 17295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:24.978987 17295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:24.979135 17295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:24.979223 17295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:25.014533 17295 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:33045
I20260620 15:03:25.014622 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:33045 every 8 connection(s)
I20260620 15:03:25.016006 17295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:03:25.020906 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.019079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:50943)
0620 15:03:25.019797 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.020009 (+ 212us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":277,"threads_started":1}
W20260620 15:03:25.021667 17457 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:34541 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.62:34541: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
I20260620 15:03:25.022503 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.022196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:38247)
0620 15:03:25.022333 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.022398 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 15:03:25.023301 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.022996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:37823)
0620 15:03:25.023119 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.023194 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 15:03:25.023547 17457 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:03:25.026353 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 17295
I20260620 15:03:25.026994 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:42041
--local_ip_for_outbound_sockets=127.2.70.2
--tserver_master_addrs=127.2.70.62:34541
--webserver_port=37347
--webserver_interface=127.2.70.2
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:25.364886 17460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:25.365176 17460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:25.365274 17460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:25.372741 17460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:25.372851 17460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:25.372965 17460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:03:25.383018 17460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:42041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=37347
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:25.384320 17460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:25.385679 17460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:25.399195 17465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:25.399289 17466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:25.399233 17468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:25.400264 17460 server_base.cc:1034] running on GCE node
I20260620 15:03:25.400911 17460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:25.402001 17460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:25.403239 17460 hybrid_clock.cc:648] HybridClock initialized: now 1781967805403186 us; error 45 us; skew 500 ppm
I20260620 15:03:25.403642 17460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:25 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967805, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:25.406821 17460 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:03:25.408337 17460 webserver.cc:460] Webserver started at http://127.2.70.2:37347/ using document root <none> and password file <none>
I20260620 15:03:25.409108 17460 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:25.409250 17460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:25.414609 17460 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:03:25.417181 17475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:25.418552 17460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:03:25.418756 17460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "792d15b325a043c68dc06417cf438b11"
format_stamp: "Formatted at 2026-06-20 15:03:19 on dist-test-slave-n5vv"
server_key: "c833e35d9535b10c7bec484b99cb1634"
server_key_iv: "3029a71d462228c6eb02c096765963c4"
server_key_version: "encryptionkey@0"
I20260620 15:03:25.419762 17460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:25.452242 17460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:25.458941 17460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:25.459242 17460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:25.460300 17460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:25.461716 17460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:25.461826 17460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:25.461956 17460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:25.462052 17460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:25.494426 17460 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:42041
I20260620 15:03:25.494510 17587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:42041 every 8 connection(s)
I20260620 15:03:25.496023 17460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:03:25.500509 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 17460
I20260620 15:03:25.501142 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:34859
--local_ip_for_outbound_sockets=127.2.70.3
--tserver_master_addrs=127.2.70.62:34541
--webserver_port=40015
--webserver_interface=127.2.70.3
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:03:25.501230 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.499222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:51581)
0620 15:03:25.499987 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.500224 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":297,"threads_started":1}
W20260620 15:03:25.501932 17588 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:34541 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.62:34541: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
I20260620 15:03:25.502871 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.502554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:34821)
0620 15:03:25.502699 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.502768 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 15:03:25.503819 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.503482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:44761)
0620 15:03:25.503607 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.503678 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 15:03:25.503997 17588 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:03:25.826306 17591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:25.826642 17591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:25.826818 17591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:25.834250 17591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:25.834417 17591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:25.834609 17591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:03:25.844666 17591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:34859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=40015
--tserver_master_addrs=127.2.70.62:34541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:25.846040 17591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:25.847368 17591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:25.860695 17596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:25.860666 17597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:25.860719 17599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:25.861423 17591 server_base.cc:1034] running on GCE node
I20260620 15:03:25.862049 17591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:25.862920 17591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:25.864115 17591 hybrid_clock.cc:648] HybridClock initialized: now 1781967805864053 us; error 55 us; skew 500 ppm
I20260620 15:03:25.864384 17591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:25 dist-test-slave-n5vv krb5kdc[16144](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967805, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:25.867544 17591 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:03:25.869063 17591 webserver.cc:460] Webserver started at http://127.2.70.3:40015/ using document root <none> and password file <none>
I20260620 15:03:25.869796 17591 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:25.869897 17591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:25.875083 17591 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:03:25.877733 17606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:25.879120 17591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:25.879253 17591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "401c08ee81bf452097fc53c65dd88a4f"
format_stamp: "Formatted at 2026-06-20 15:03:20 on dist-test-slave-n5vv"
server_key: "bbbb344b2895750e2783ab940ba9eb32"
server_key_iv: "590755bc09a6271a5a7e58e18fc10523"
server_key_version: "encryptionkey@0"
I20260620 15:03:25.880193 17591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:25.892693 17591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:25.896615 17591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:25.896840 17591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:25.897786 17591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:25.899037 17591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:25.899147 17591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:25.899217 17591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:25.899282 17591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:25.933079 17591 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:34859
I20260620 15:03:25.933180 17718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:34859 every 8 connection(s)
I20260620 15:03:25.934579 17591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:03:25.939291 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.937411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:53725)
0620 15:03:25.938192 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.938383 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":280,"threads_started":1}
W20260620 15:03:25.940025 17719 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:34541 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.62:34541: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
I20260620 15:03:25.940827 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.940524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:40069)
0620 15:03:25.940663 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.940726 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:03:25.941700 17721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:25.941360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:34605)
0620 15:03:25.941520 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:25.941598 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:03:25.941985 17719 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:03:25.944175 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 17591
I20260620 15:03:26.024859 17723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:26.024024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:59661)
0620 15:03:26.024541 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:26.024626 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":259,"threads_started":1}
I20260620 15:03:26.505283 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:26.504516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:38007)
0620 15:03:26.504961 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:26.505060 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":220,"threads_started":1}
I20260620 15:03:26.943325 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:26.942494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:35117)
0620 15:03:26.942961 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:26.943101 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":214,"threads_started":1}
I20260620 15:03:27.026466 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:27.025742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:55927)
0620 15:03:27.026214 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:27.026300 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":241,"threads_started":1}
I20260620 15:03:27.506901 17727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:27.506139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:43783)
0620 15:03:27.506653 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:27.506739 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":245,"threads_started":1}
I20260620 15:03:27.944980 17728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:27.944247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:42209)
0620 15:03:27.944735 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:27.944820 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
I20260620 15:03:28.028119 17729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:28.027395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:34595)
0620 15:03:28.027855 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:28.027942 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":235,"threads_started":1}
I20260620 15:03:28.508939 17730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:28.507880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:40647)
0620 15:03:28.508510 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:28.508653 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":324,"threads_started":1}
I20260620 15:03:28.946799 17731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:28.946002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:37501)
0620 15:03:28.946504 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:28.946608 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":252,"threads_started":1}
I20260620 15:03:29.029898 17732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:29.029040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:51943)
0620 15:03:29.029603 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:29.029696 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":244,"threads_started":1}
I20260620 15:03:29.510818 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:29.510066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:40305)
0620 15:03:29.510563 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:29.510647 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":256,"threads_started":1}
I20260620 15:03:29.948395 17734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:29.947710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:54487)
0620 15:03:29.948130 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:29.948205 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260620 15:03:30.031599 17735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:30.030845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:60887)
0620 15:03:30.031340 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:30.031425 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":262,"threads_started":1}
I20260620 15:03:30.512786 17736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:30.511834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:58951)
0620 15:03:30.512432 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:30.512545 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":302,"threads_started":1}
I20260620 15:03:30.950222 17737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:30.949408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:55905)
0620 15:03:30.949955 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:30.950038 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":310,"threads_started":1}
I20260620 15:03:31.033591 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:31.032740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:55747)
0620 15:03:31.033253 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:31.033351 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":263,"threads_started":1}
I20260620 15:03:31.514627 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:31.513851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:55271)
0620 15:03:31.514323 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:31.514423 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":243,"threads_started":1}
I20260620 15:03:31.951802 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:31.951100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:36953)
0620 15:03:31.951542 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:31.951637 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260620 15:03:32.035581 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:32.034595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:54015)
0620 15:03:32.035212 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:32.035314 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":244,"threads_started":1}
I20260620 15:03:32.516804 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:32.515767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:55349)
0620 15:03:32.516464 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:32.516585 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":271,"threads_started":1}
I20260620 15:03:32.953629 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:32.952702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:56099)
0620 15:03:32.953221 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:32.953317 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":235,"threads_started":1}
I20260620 15:03:33.037465 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:33.036739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:59305)
0620 15:03:33.037193 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:33.037273 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":232,"threads_started":1}
I20260620 15:03:33.518829 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:33.517984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:49511)
0620 15:03:33.518534 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:33.518626 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":262,"threads_started":1}
I20260620 15:03:33.955359 17746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:33.954585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:47311)
0620 15:03:33.955024 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:33.955164 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260620 15:03:34.039284 17747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:34.038467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:47375)
0620 15:03:34.039004 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:34.039096 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":284,"threads_started":1}
I20260620 15:03:34.520800 17748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:34.520009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:39981)
0620 15:03:34.520525 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:34.520620 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":226,"threads_started":1}
I20260620 15:03:34.957118 17749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:34.956314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:37817)
0620 15:03:34.956836 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:34.956936 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":264,"threads_started":1}
I20260620 15:03:35.040973 17750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:35.040250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:38709)
0620 15:03:35.040725 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:35.040808 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
I20260620 15:03:35.522549 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:35.521839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:49253)
0620 15:03:35.522300 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:35.522384 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":243,"threads_started":1}
I20260620 15:03:35.958870 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:35.958152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:59731)
0620 15:03:35.958622 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:35.958708 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":243,"threads_started":1}
I20260620 15:03:36.042706 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:36.041984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:43097)
0620 15:03:36.042432 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:36.042524 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":225,"threads_started":1}
I20260620 15:03:36.524161 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:36.523415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:51885)
0620 15:03:36.523893 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:36.523978 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":276,"threads_started":1}
I20260620 15:03:36.960697 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:36.959928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:34553)
0620 15:03:36.960380 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:36.960480 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20260620 15:03:37.044356 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:37.043659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:54497)
0620 15:03:37.044126 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:37.044201 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":238,"threads_started":1}
I20260620 15:03:37.525820 17757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:37.525060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:59249)
0620 15:03:37.525550 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:37.525656 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260620 15:03:37.962321 17758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:37.961664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:33471)
0620 15:03:37.962095 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:37.962171 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":205,"threads_started":1}
I20260620 15:03:38.046064 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:38.045256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:52947)
0620 15:03:38.045762 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:38.045848 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":291,"threads_started":1}
I20260620 15:03:38.527416 17760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:38.526661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:47401)
0620 15:03:38.527152 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:38.527231 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":251,"threads_started":1}
I20260620 15:03:38.963939 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:38.963208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:32815)
0620 15:03:38.963666 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:38.963752 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":203,"threads_started":1}
I20260620 15:03:39.047703 17762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:39.046991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:38717)
0620 15:03:39.047415 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:39.047497 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":227,"threads_started":1}
I20260620 15:03:39.528975 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:39.528297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:50029)
0620 15:03:39.528717 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:39.528797 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":232,"threads_started":1}
I20260620 15:03:39.965567 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:39.964795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:50159)
0620 15:03:39.965254 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:39.965336 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":236,"threads_started":1}
I20260620 15:03:40.049597 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:40.048820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:34607)
0620 15:03:40.049281 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:40.049359 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":231,"threads_started":1}
I20260620 15:03:40.531026 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:40.530100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.2:43025)
0620 15:03:40.530670 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:40.530755 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":326,"threads_started":1}
I20260620 15:03:40.951821 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 17295
I20260620 15:03:40.967494 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:40.966768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.3:38053)
0620 15:03:40.967242 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:40.967319 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260620 15:03:40.971426 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:40.953082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.2.70.1:33045)
0620 15:03:40.953748 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:40.953770 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:40.953822 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:03:40.955891 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:40.955909 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:40.956417 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:40.956785 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:40.958385 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:40.959741 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:40.961063 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:40.961836 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:40.962037 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:40.964749 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:40.964757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:40.964774 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:40.964816 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:40.968161 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:40.968863 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:40.968870 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:40.968877 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:40.969046 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:40.969429 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:40.969434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:40.969436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:40.970010 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:40.970314 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:40.970737 (+ 423us) server_negotiation.cc:299] Negotiation successful
0620 15:03:40.971163 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":267,"threads_started":1}
I20260620 15:03:40.972796 17452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:41.057531 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:41.056331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34541 (local address 127.2.70.1:43845)
0620 15:03:41.056931 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:41.057135 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":285,"threads_started":1}
I20260620 15:03:41.057732 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 17295
I20260620 15:03:41.074962 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 17460
I20260620 15:03:41.093339 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:41.075881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44048 (local address 127.2.70.2:42041)
0620 15:03:41.076423 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:41.076441 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:41.076490 (+ 49us) server_negotiation.cc:407] Connection header received
0620 15:03:41.078130 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:41.078139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:41.078751 (+ 612us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:41.079076 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:41.080428 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:41.081769 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:41.082967 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:41.083720 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:41.083860 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:41.086816 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:41.086824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:41.086839 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:41.086887 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:41.089787 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:41.090665 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:41.090672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:41.090706 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:41.090932 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:41.091331 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:41.091337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:41.091340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:41.091873 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:41.092186 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:41.092634 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 15:03:41.093063 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":271,"threads_started":1}
I20260620 15:03:41.094753 17583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:41.181870 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 17460
I20260620 15:03:41.199309 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 17591
I20260620 15:03:41.217010 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:41.200238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.2.70.3:34859)
0620 15:03:41.200929 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:41.200944 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:41.200989 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:03:41.202853 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:41.202865 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:41.203334 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:41.203643 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:41.204881 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:41.206257 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:41.207560 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:41.208228 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:41.208363 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:41.211044 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:41.211052 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:41.211067 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:41.211108 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:41.214054 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:41.214684 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:41.214691 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:41.214698 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:41.214961 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:41.215307 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:41.215310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:41.215312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:41.215813 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:41.216051 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:41.216372 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 15:03:41.216690 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":377,"threads_started":1}
I20260620 15:03:41.218443 17714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:41.301476 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 17591
2026-06-20T15:03:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24635 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:03:41 dist-test-slave-n5vv krb5kdc[17777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:03:41 dist-test-slave-n5vv krb5kdc[17777](info): set up 2 sockets
Jun 20 15:03:41 dist-test-slave-n5vv krb5kdc[17777](info): commencing operation
krb5kdc: starting...
W20260620 15:03:42.414264 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:03:42 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:03:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:03:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:03:42.652654 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34899
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:42.978785 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:42.979076 17790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:42.979166 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:42.986306 17790 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:03:42.986410 17790 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:03:42.986449 17790 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:03:42.986500 17790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:03:42.986562 17790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:42.986630 17790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:42.986675 17790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:03:42.986723 17790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:03:42.997037 17790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:42.998333 17790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:42.999626 17790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:43.011291 17796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:43.011325 17798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:43.011260 17795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:43.011855 17790 server_base.cc:1034] running on GCE node
I20260620 15:03:43.012789 17790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:43.014248 17790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:43.015488 17790 hybrid_clock.cc:648] HybridClock initialized: now 1781967823015403 us; error 67 us; skew 500 ppm
I20260620 15:03:43.015826 17790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:43 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:43.019119 17790 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:03:43.020567 17790 webserver.cc:460] Webserver started at http://127.2.70.62:45599/ using document root <none> and password file <none>
I20260620 15:03:43.021274 17790 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:43.021409 17790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:43.021760 17790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:43.024812 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "bdce2c57c49e57bcf476c1e4cf923fc1"
server_key_iv: "24980c68bfdfec4560ae8f6086e2df23"
server_key_version: "encryptionkey@0"
I20260620 15:03:43.025696 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "bdce2c57c49e57bcf476c1e4cf923fc1"
server_key_iv: "24980c68bfdfec4560ae8f6086e2df23"
server_key_version: "encryptionkey@0"
I20260620 15:03:43.030843 17790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 15:03:43.034250 17805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:43.035476 17790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:03:43.035682 17790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "bdce2c57c49e57bcf476c1e4cf923fc1"
server_key_iv: "24980c68bfdfec4560ae8f6086e2df23"
server_key_version: "encryptionkey@0"
I20260620 15:03:43.035902 17790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:43.056080 17790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:43.060055 17790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:43.060319 17790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:43.078374 17790 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:34899
I20260620 15:03:43.078395 17856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:34899 every 8 connection(s)
I20260620 15:03:43.079725 17790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:03:43.083389 17857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:43.083990 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 17790
I20260620 15:03:43.084214 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:03:43.084865 2328 external_mini_cluster.cc:1428] Setting key 97e4067deeb47d96de5cebcee5b815eb
I20260620 15:03:43.097082 17857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Bootstrap starting.
Jun 20 15:03:43 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:43.100795 17857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:43.102222 17857 log.cc:824] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:43.105458 17857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: No bootstrap required, opened a new log
I20260620 15:03:43.108974 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:43.086925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.2.70.62:34899)
0620 15:03:43.087752 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:43.087762 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:43.087802 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:03:43.089427 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:43.089447 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:43.089889 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:43.090242 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:43.091946 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:43.093198 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:43.095265 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:43.095903 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:43.096033 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:43.100678 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:43.100691 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:43.100711 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:43.100766 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:43.104546 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:43.105202 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:43.105210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:43.105216 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:43.105413 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:43.105838 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:43.105843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:43.105846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:43.106543 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:43.106849 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:43.107163 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 15:03:43.107633 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":386,"threads_started":1}
I20260620 15:03:43.115759 17857 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:03:43.116122 17857 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:43.116199 17857 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa95e4ac5f344d2aab0f7b63bc1d720e, State: Initialized, Role: FOLLOWER
I20260620 15:03:43.116716 17857 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:03:43.116942 17857 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:03:43.117046 17857 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:03:43.117179 17857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:43.119807 17857 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:03:43.120272 17857 leader_election.cc:302] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa95e4ac5f344d2aab0f7b63bc1d720e; no voters:
I20260620 15:03:43.120812 17857 leader_election.cc:288] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:03:43.120980 17862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:43.121665 17862 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 1 LEADER]: Becoming Leader. State: Replica: fa95e4ac5f344d2aab0f7b63bc1d720e, State: Running, Role: LEADER
I20260620 15:03:43.122195 17862 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:03:43.123045 17857 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:03:43.125006 17864 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa95e4ac5f344d2aab0f7b63bc1d720e. Latest consensus state: current_term: 1 leader_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } } }
I20260620 15:03:43.125320 17864 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:43.125689 17863 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } } }
I20260620 15:03:43.126142 17863 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: This master's current role is: LEADER
I20260620 15:03:43.129733 17869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:03:43.134995 17869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:03:43.143935 17869 catalog_manager.cc:1347] Generated new cluster ID: f340bedf6e9f4545a84d6d977ba35575
I20260620 15:03:43.144049 17869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:03:43.173074 17869 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:03:43.175223 17869 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:03:43.193694 17869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Generated new TSK 0
I20260620 15:03:43.194635 17869 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:03:43.262118 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34899
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:43.597200 17883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:43.597628 17883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:43.597810 17883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:43.605109 17883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:43.605279 17883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:43.605517 17883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:03:43.615831 17883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:43.617187 17883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:43.618588 17883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:43.632629 17889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:43.632525 17891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:43.632669 17888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:43.633559 17883 server_base.cc:1034] running on GCE node
I20260620 15:03:43.634152 17883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:43.635085 17883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:43.636283 17883 hybrid_clock.cc:648] HybridClock initialized: now 1781967823636242 us; error 32 us; skew 500 ppm
I20260620 15:03:43.636617 17883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:43 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:43.639850 17883 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:03:43.641541 17883 webserver.cc:460] Webserver started at http://127.2.70.1:43285/ using document root <none> and password file <none>
I20260620 15:03:43.642225 17883 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:43.642351 17883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:43.642668 17883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:43.645602 17883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "8299f9a26ec44f97a65718498e2992f4"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "16901be973986c7a58999594d1eaf295"
server_key_iv: "a665a3bad7e76beb651e127657cc6453"
server_key_version: "encryptionkey@0"
I20260620 15:03:43.646363 17883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "8299f9a26ec44f97a65718498e2992f4"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "16901be973986c7a58999594d1eaf295"
server_key_iv: "a665a3bad7e76beb651e127657cc6453"
server_key_version: "encryptionkey@0"
I20260620 15:03:43.651376 17883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:03:43.654831 17898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:43.656204 17883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:03:43.656376 17883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "8299f9a26ec44f97a65718498e2992f4"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "16901be973986c7a58999594d1eaf295"
server_key_iv: "a665a3bad7e76beb651e127657cc6453"
server_key_version: "encryptionkey@0"
I20260620 15:03:43.656580 17883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:43.679780 17883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:43.685452 17883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:43.685829 17883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:43.686892 17883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:43.688251 17883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:43.688333 17883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:43.688494 17883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:43.688570 17883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:43.721972 17883 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:40489
I20260620 15:03:43.722059 18010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:40489 every 8 connection(s)
I20260620 15:03:43.723305 17883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:03:43.727921 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 17883
I20260620 15:03:43.728155 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:03:43.729012 2328 external_mini_cluster.cc:1428] Setting key 3cba31c359b2465072b3bfbefbc0d8bf
Jun 20 15:03:43 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967823, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:43.747985 18013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:43.726227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:45603 (local address 127.2.70.62:34899)
0620 15:03:43.726665 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:43.726669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:43.727647 (+ 978us) server_negotiation.cc:407] Connection header received
0620 15:03:43.730315 (+ 2668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:43.730320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:43.730450 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:43.730594 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:43.732857 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:43.733745 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:43.736333 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:43.736770 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:43.736874 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:43.741162 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:43.741171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:43.741179 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:43.741224 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:43.744753 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:43.745449 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:43.745457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:43.745463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:43.745635 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:43.746039 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:43.746046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:43.746052 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:43.746417 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:43.746557 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:43.747242 (+ 685us) server_negotiation.cc:299] Negotiation successful
0620 15:03:43.747500 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":237,"threads_started":1}
I20260620 15:03:43.748613 18014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:43.726704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:45603)
0620 15:03:43.727453 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:43.727507 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:43.729960 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:43.730790 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:43.730798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:43.731646 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:43.732514 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:43.732543 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:43.733924 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:43.733932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:43.736153 (+ 2221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:43.736172 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:43.737017 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:43.737026 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:43.737371 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:43.738156 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:43.738191 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:43.740921 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:43.744887 (+ 3966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:43.744894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:43.744908 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:43.745331 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:43.745737 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:43.745740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:43.745743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:43.745922 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:43.746540 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:43.746545 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:43.746866 (+ 321us) client_negotiation.cc:769] Sending connection context
0620 15:03:43.747107 (+ 241us) client_negotiation.cc:240] Negotiation successful
0620 15:03:43.747495 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":276,"threads_started":1}
I20260620 15:03:43.750238 18011 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34899
I20260620 15:03:43.750578 18011 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:43.751456 18011 heartbeater.cc:511] Master 127.2.70.62:34899 requested a full tablet report, sending...
I20260620 15:03:43.753851 17822 ts_manager.cc:194] Registered new tserver with Master: 8299f9a26ec44f97a65718498e2992f4 (127.2.70.1:40489)
I20260620 15:03:43.755287 17822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:45603
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:03:43.784842 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34899
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:44.115389 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:44.115664 18017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:44.115834 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:44.123136 18017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:44.123266 18017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:44.123440 18017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:03:44.133635 18017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:44.134922 18017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:44.136330 18017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:44.150269 18025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:44.150291 18023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:44.150378 18022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:44.151216 18017 server_base.cc:1034] running on GCE node
I20260620 15:03:44.151863 18017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:44.152854 18017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:44.154105 18017 hybrid_clock.cc:648] HybridClock initialized: now 1781967824154024 us; error 65 us; skew 500 ppm
I20260620 15:03:44.154458 18017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:44 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:44.157621 18017 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:03:44.159056 18017 webserver.cc:460] Webserver started at http://127.2.70.2:38797/ using document root <none> and password file <none>
I20260620 15:03:44.159850 18017 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:44.160002 18017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:44.160313 18017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:44.163379 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "5105d1414cb245b79dfbdf04d09cb863"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "982282d0e55b287d7f58bfa2816b20cb"
server_key_iv: "811f3c78c14f83464710b70f1fd7b13d"
server_key_version: "encryptionkey@0"
I20260620 15:03:44.164147 18017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "5105d1414cb245b79dfbdf04d09cb863"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "982282d0e55b287d7f58bfa2816b20cb"
server_key_iv: "811f3c78c14f83464710b70f1fd7b13d"
server_key_version: "encryptionkey@0"
I20260620 15:03:44.169384 18017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:03:44.172741 18032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:44.174036 18017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:03:44.174207 18017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "5105d1414cb245b79dfbdf04d09cb863"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "982282d0e55b287d7f58bfa2816b20cb"
server_key_iv: "811f3c78c14f83464710b70f1fd7b13d"
server_key_version: "encryptionkey@0"
I20260620 15:03:44.174394 18017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:44.204454 18017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:44.208756 18017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:44.209052 18017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:44.210079 18017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:44.211421 18017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:44.211522 18017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:44.211666 18017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:44.211742 18017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:44.245535 18017 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:39563
I20260620 15:03:44.245604 18144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:39563 every 8 connection(s)
I20260620 15:03:44.246906 18017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:03:44.247254 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 18017
I20260620 15:03:44.247433 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:03:44.248073 2328 external_mini_cluster.cc:1428] Setting key b208a8facf71025755729588ab410ae1
Jun 20 15:03:44 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:44.271324 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.249708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:48253 (local address 127.2.70.62:34899)
0620 15:03:44.250178 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:44.250184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:44.251387 (+ 1203us) server_negotiation.cc:407] Connection header received
0620 15:03:44.254289 (+ 2902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:44.254293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:44.254357 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:44.254448 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:44.256644 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.257374 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.259680 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.260411 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.260563 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.264016 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:44.264025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:44.264033 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:44.264077 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:44.267895 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:44.268689 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:44.268697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:44.268703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:44.268865 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:44.269286 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:44.269294 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:44.269300 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:44.269746 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:44.269909 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:44.270600 (+ 691us) server_negotiation.cc:299] Negotiation successful
0620 15:03:44.270882 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":250,"threads_started":1}
I20260620 15:03:44.272100 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.250382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:48253)
0620 15:03:44.251214 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:44.251263 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:44.253992 (+ 2729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:44.254724 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:44.254740 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:44.255534 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:44.256449 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.256479 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.257555 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.257564 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.259506 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.259517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.260643 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.260649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.260846 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.261644 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:44.261665 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:44.263788 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:44.268061 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:44.268069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:44.268083 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:44.268548 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:44.268970 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:44.268974 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:44.268976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:44.269160 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:44.269868 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:44.269874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:44.270211 (+ 337us) client_negotiation.cc:769] Sending connection context
0620 15:03:44.270437 (+ 226us) client_negotiation.cc:240] Negotiation successful
0620 15:03:44.270975 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
I20260620 15:03:44.273650 18145 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34899
I20260620 15:03:44.273999 18145 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:03:44.274940 18145 heartbeater.cc:511] Master 127.2.70.62:34899 requested a full tablet report, sending...
I20260620 15:03:44.276548 17822 ts_manager.cc:194] Registered new tserver with Master: 5105d1414cb245b79dfbdf04d09cb863 (127.2.70.2:39563)
I20260620 15:03:44.277345 17822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:48253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:03:44.304852 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34899
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:03:44.635596 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:03:44.635861 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:03:44.636039 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:03:44.643416 18151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:03:44.643543 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:03:44.643733 18151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:03:44.653273 18151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:03:44.654606 18151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:03:44.655963 18151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:03:44.669152 18157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:44.669107 18156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:03:44.669189 18159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:03:44.670089 18151 server_base.cc:1034] running on GCE node
I20260620 15:03:44.670693 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:03:44.671602 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:03:44.672797 18151 hybrid_clock.cc:648] HybridClock initialized: now 1781967824672742 us; error 60 us; skew 500 ppm
I20260620 15:03:44.673126 18151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:03:44 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:03:44.676323 18151 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:03:44.678016 18151 webserver.cc:460] Webserver started at http://127.2.70.3:39259/ using document root <none> and password file <none>
I20260620 15:03:44.678853 18151 fs_manager.cc:359] Metadata directory not provided
I20260620 15:03:44.678979 18151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:03:44.679333 18151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:03:44.682420 18151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "9a29005a18a84af99ad1f61aed3c0dfa"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "0e8b5de7671595e8650a31106e30497b"
server_key_iv: "de81e59b6b90abca84a6fc7616cf79c4"
server_key_version: "encryptionkey@0"
I20260620 15:03:44.683234 18151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "9a29005a18a84af99ad1f61aed3c0dfa"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "0e8b5de7671595e8650a31106e30497b"
server_key_iv: "de81e59b6b90abca84a6fc7616cf79c4"
server_key_version: "encryptionkey@0"
I20260620 15:03:44.689535 18151 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260620 15:03:44.692941 18166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:44.694334 18151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:03:44.694494 18151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "9a29005a18a84af99ad1f61aed3c0dfa"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "0e8b5de7671595e8650a31106e30497b"
server_key_iv: "de81e59b6b90abca84a6fc7616cf79c4"
server_key_version: "encryptionkey@0"
I20260620 15:03:44.694700 18151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:03:44.710939 18151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:03:44.715109 18151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:03:44.715394 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:03:44.716326 18151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:03:44.717658 18151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:03:44.717729 18151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:44.717878 18151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:03:44.717932 18151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:03:44.751881 18151 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:34629
I20260620 15:03:44.751926 18278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:34629 every 8 connection(s)
I20260620 15:03:44.753229 18151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:03:44.757457 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 18151
I20260620 15:03:44.757697 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:03:44.758270 2328 external_mini_cluster.cc:1428] Setting key 24a177cd4d3fbfc24f201b3a441a6351
I20260620 15:03:44.758374 18011 heartbeater.cc:503] Master 127.2.70.62:34899 was elected leader, sending a full tablet report...
Jun 20 15:03:44 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967824, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:03:44.773613 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.755815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:39009 (local address 127.2.70.62:34899)
0620 15:03:44.755996 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:44.756001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:44.757196 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 15:03:44.759815 (+ 2619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:44.759819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:44.759894 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:44.759996 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:44.761833 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.762534 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.763970 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.764341 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.764432 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.767764 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:44.767770 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:44.767773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:44.767812 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:44.770338 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:44.771087 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:44.771091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:44.771094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:44.771216 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:44.771615 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:44.771618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:44.771620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:44.771941 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:44.772099 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:44.773028 (+ 929us) server_negotiation.cc:299] Negotiation successful
0620 15:03:44.773255 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:03:44.774632 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.756294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:39009)
0620 15:03:44.756982 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:44.757059 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:44.759434 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:44.760309 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:44.760321 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:44.760891 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:03:44.761687 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.761707 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.762663 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.762667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.763855 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.763866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.764522 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.764528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.764722 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.765379 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:03:44.765401 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:03:44.767560 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:03:44.770537 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:44.770545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:44.770567 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:44.770972 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:44.771300 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:03:44.771303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:03:44.771305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:03:44.771476 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:03:44.772083 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:03:44.772089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:03:44.772536 (+ 447us) client_negotiation.cc:769] Sending connection context
0620 15:03:44.772896 (+ 360us) client_negotiation.cc:240] Negotiation successful
0620 15:03:44.773408 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":268,"threads_started":1}
I20260620 15:03:44.775929 18279 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34899
I20260620 15:03:44.776211 18279 heartbeater.cc:464] Registering TS with master...
I20260620 15:03:44.776906 18279 heartbeater.cc:511] Master 127.2.70.62:34899 requested a full tablet report, sending...
I20260620 15:03:44.778394 17822 ts_manager.cc:194] Registered new tserver with Master: 9a29005a18a84af99ad1f61aed3c0dfa (127.2.70.3:34629)
I20260620 15:03:44.779247 17822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:39009
I20260620 15:03:44.786048 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:03:44.802659 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.790556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.2.70.62:34899)
0620 15:03:44.790718 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:44.790723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:44.791149 (+ 426us) server_negotiation.cc:407] Connection header received
0620 15:03:44.791281 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:44.791284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:44.791339 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:44.791419 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:44.792787 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.793615 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.794874 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.795244 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.795331 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.797761 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:44.797765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:44.797769 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:44.797797 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:44.800380 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:44.801041 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:44.801045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:44.801047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:44.801174 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:44.801524 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:44.801527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:44.801529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:44.801827 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:44.802006 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:44.802251 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 15:03:44.802428 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:03:44.806702 17822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:03:44.813452 17822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:03:44.849788 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.825903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34629 (local address 127.0.0.1:44936)
0620 15:03:44.840137 (+ 14234us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:44.840187 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:44.840490 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:44.841659 (+ 1169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:44.841667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:44.841698 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:44.842151 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.842170 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.844674 (+ 2504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.844679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.846759 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.846771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.848681 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.848690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.848889 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.848950 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 15:03:44.849072 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 15:03:44.849284 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13927,"mutex_wait_us":1,"thread_start_us":13791,"threads_started":1}
I20260620 15:03:44.849874 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.826336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:39563 (local address 127.0.0.1:35692)
0620 15:03:44.841413 (+ 15077us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:44.841438 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:44.841619 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:44.843204 (+ 1585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:44.843211 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:44.843240 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:44.843727 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.843737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.845378 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.845385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.847429 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.847441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.849004 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.849010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.849221 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.849255 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:03:44.849318 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 15:03:44.849450 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14889,"mutex_wait_us":3,"thread_start_us":15195,"threads_started":1}
I20260620 15:03:44.850494 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.826217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44936 (local address 127.2.70.3:34629)
0620 15:03:44.839425 (+ 13208us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:44.839438 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:44.840324 (+ 886us) server_negotiation.cc:407] Connection header received
0620 15:03:44.840781 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:44.840790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:44.841060 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:44.841246 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:44.842398 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.844503 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.846912 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.848449 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.849590 (+ 1141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.849847 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:44.849945 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 15:03:44.850126 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13021,"thread_start_us":12673,"threads_started":1}
I20260620 15:03:44.851272 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.826540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35692 (local address 127.2.70.2:39563)
0620 15:03:44.842529 (+ 15989us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:44.842539 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:44.842566 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:03:44.842670 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:44.842678 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:44.842920 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:44.843123 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:44.843906 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.845227 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.847599 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.848770 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.850001 (+ 1231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.850751 (+ 750us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:44.850830 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 15:03:44.850983 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15809,"thread_start_us":15652,"threads_started":1}
I20260620 15:03:44.851822 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.841920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:46948)
0620 15:03:44.842871 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:44.842894 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:44.843036 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:44.844505 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:44.844510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:44.844537 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:44.845016 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.845028 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.847010 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.847015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.849132 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.849144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.851006 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.851013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.851187 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.851225 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 15:03:44.851313 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 15:03:44.851437 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"mutex_wait_us":24,"thread_start_us":1367,"threads_started":1}
I20260620 15:03:44.853091 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.842359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.2.70.1:40489)
0620 15:03:44.844008 (+ 1649us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:44.844016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:44.844041 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:03:44.844129 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:44.844137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:44.844319 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:44.844506 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:44.845227 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.846822 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.849300 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.850820 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.850961 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.851195 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:44.851369 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 15:03:44.852751 (+ 1382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1492,"thread_start_us":1330,"threads_started":1}
I20260620 15:03:44.854476 18214 tablet_service.cc:1467] Processing CreateTablet for tablet f7335b09d74347ac84fa38ffe30b8b92 (DEFAULT_TABLE table=test-table [id=eb0a18cb5fe441c6bfa55fa2f41f3888]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:44.854918 18080 tablet_service.cc:1467] Processing CreateTablet for tablet f7335b09d74347ac84fa38ffe30b8b92 (DEFAULT_TABLE table=test-table [id=eb0a18cb5fe441c6bfa55fa2f41f3888]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:44.855779 18214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7335b09d74347ac84fa38ffe30b8b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:44.856312 18080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7335b09d74347ac84fa38ffe30b8b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:44.856551 17946 tablet_service.cc:1467] Processing CreateTablet for tablet f7335b09d74347ac84fa38ffe30b8b92 (DEFAULT_TABLE table=test-table [id=eb0a18cb5fe441c6bfa55fa2f41f3888]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:03:44.857924 17946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7335b09d74347ac84fa38ffe30b8b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:03:44.867760 18297 tablet_bootstrap.cc:492] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Bootstrap starting.
I20260620 15:03:44.870828 18298 tablet_bootstrap.cc:492] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Bootstrap starting.
I20260620 15:03:44.871683 18297 tablet_bootstrap.cc:654] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:44.872720 18299 tablet_bootstrap.cc:492] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Bootstrap starting.
I20260620 15:03:44.873014 18297 log.cc:824] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:44.875310 18298 tablet_bootstrap.cc:654] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:44.875988 18299 tablet_bootstrap.cc:654] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Neither blocks nor log segments found. Creating new log.
I20260620 15:03:44.876336 18297 tablet_bootstrap.cc:492] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: No bootstrap required, opened a new log
I20260620 15:03:44.876538 18298 log.cc:824] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:44.876642 18297 ts_tablet_manager.cc:1397] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260620 15:03:44.877115 18299 log.cc:824] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:03:44.879545 18298 tablet_bootstrap.cc:492] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: No bootstrap required, opened a new log
I20260620 15:03:44.879828 18298 ts_tablet_manager.cc:1397] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260620 15:03:44.880369 18299 tablet_bootstrap.cc:492] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: No bootstrap required, opened a new log
I20260620 15:03:44.880596 18299 ts_tablet_manager.cc:1397] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 15:03:44.884747 18297 raft_consensus.cc:348] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.885123 18297 raft_consensus.cc:374] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:44.885241 18297 raft_consensus.cc:729] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a29005a18a84af99ad1f61aed3c0dfa, State: Initialized, Role: FOLLOWER
I20260620 15:03:44.885876 18297 consensus_queue.cc:260] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.886945 18279 heartbeater.cc:503] Master 127.2.70.62:34899 was elected leader, sending a full tablet report...
I20260620 15:03:44.887637 18297 ts_tablet_manager.cc:1428] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
I20260620 15:03:44.890686 18298 raft_consensus.cc:348] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.891199 18298 raft_consensus.cc:374] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:44.891347 18298 raft_consensus.cc:729] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5105d1414cb245b79dfbdf04d09cb863, State: Initialized, Role: FOLLOWER
I20260620 15:03:44.891211 18299 raft_consensus.cc:348] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.891788 18299 raft_consensus.cc:374] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:03:44.891933 18299 raft_consensus.cc:729] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8299f9a26ec44f97a65718498e2992f4, State: Initialized, Role: FOLLOWER
I20260620 15:03:44.892040 18298 consensus_queue.cc:260] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.892554 18299 consensus_queue.cc:260] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.893312 18145 heartbeater.cc:503] Master 127.2.70.62:34899 was elected leader, sending a full tablet report...
I20260620 15:03:44.894042 18298 ts_tablet_manager.cc:1428] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260620 15:03:44.894773 18299 ts_tablet_manager.cc:1428] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Time spent starting tablet: real 0.014s user 0.014s sys 0.002s
I20260620 15:03:44.935902 18305 raft_consensus.cc:482] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:03:44.936183 18305 raft_consensus.cc:504] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.937932 18305 leader_election.cc:288] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a29005a18a84af99ad1f61aed3c0dfa (127.2.70.3:34629), 5105d1414cb245b79dfbdf04d09cb863 (127.2.70.2:39563)
I20260620 15:03:44.947605 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.938444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:38403 (local address 127.2.70.3:34629)
0620 15:03:44.938654 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:44.938661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:44.939506 (+ 845us) server_negotiation.cc:407] Connection header received
0620 15:03:44.939733 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:44.939738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:44.939820 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:44.939912 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:44.940855 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.942039 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.945064 (+ 3025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.946360 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.946497 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.946604 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:44.947187 (+ 583us) server_negotiation.cc:299] Negotiation successful
0620 15:03:44.947332 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:03:44.947606 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.938444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:49991 (local address 127.2.70.2:39563)
0620 15:03:44.938656 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:44.938662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:44.939533 (+ 871us) server_negotiation.cc:407] Connection header received
0620 15:03:44.939820 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:44.939825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:44.939916 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:44.940036 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:03:44.940873 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.942046 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.945066 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.946360 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.946497 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.946610 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:44.947187 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 15:03:44.947327 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:03:44.947847 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.938426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34629 (local address 127.2.70.1:38403)
0620 15:03:44.939370 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:44.939391 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:44.939578 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:44.940079 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:44.940084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:44.940128 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:44.940663 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.940681 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.942247 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.942252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.944884 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.944900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.946661 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.946667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.946952 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.947019 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 15:03:44.947122 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:03:44.947335 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":325,"spinlock_wait_cycles":9984,"thread_start_us":445,"threads_started":1}
I20260620 15:03:44.947870 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:44.938426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:39563 (local address 127.2.70.1:49991)
0620 15:03:44.939441 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0620 15:03:44.939457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:03:44.939632 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:03:44.940079 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:03:44.940084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:03:44.940122 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:03:44.940705 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.940714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.942248 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.942252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.944884 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:03:44.944900 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:03:44.946637 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:03:44.946648 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:03:44.946946 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:44.947024 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 15:03:44.947122 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:03:44.947335 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":240,"spinlock_wait_cycles":896,"thread_start_us":549,"threads_started":1}
I20260620 15:03:44.948933 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7335b09d74347ac84fa38ffe30b8b92" candidate_uuid: "8299f9a26ec44f97a65718498e2992f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" is_pre_election: true
I20260620 15:03:44.949036 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7335b09d74347ac84fa38ffe30b8b92" candidate_uuid: "8299f9a26ec44f97a65718498e2992f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5105d1414cb245b79dfbdf04d09cb863" is_pre_election: true
I20260620 15:03:44.949506 18234 raft_consensus.cc:2454] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8299f9a26ec44f97a65718498e2992f4 in term 0.
I20260620 15:03:44.949622 18100 raft_consensus.cc:2454] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8299f9a26ec44f97a65718498e2992f4 in term 0.
I20260620 15:03:44.950253 17899 leader_election.cc:302] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5105d1414cb245b79dfbdf04d09cb863, 8299f9a26ec44f97a65718498e2992f4, 9a29005a18a84af99ad1f61aed3c0dfa; no voters:
I20260620 15:03:44.950644 18305 raft_consensus.cc:2781] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:03:44.950789 18305 raft_consensus.cc:482] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:03:44.950901 18305 raft_consensus.cc:3037] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:44.953085 18305 raft_consensus.cc:504] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.953794 18305 leader_election.cc:288] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [CANDIDATE]: Term 1 election: Requested vote from peers 9a29005a18a84af99ad1f61aed3c0dfa (127.2.70.3:34629), 5105d1414cb245b79dfbdf04d09cb863 (127.2.70.2:39563)
I20260620 15:03:44.954159 18234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7335b09d74347ac84fa38ffe30b8b92" candidate_uuid: "8299f9a26ec44f97a65718498e2992f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a29005a18a84af99ad1f61aed3c0dfa"
I20260620 15:03:44.954356 18234 raft_consensus.cc:3037] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:44.954320 18100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7335b09d74347ac84fa38ffe30b8b92" candidate_uuid: "8299f9a26ec44f97a65718498e2992f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5105d1414cb245b79dfbdf04d09cb863"
I20260620 15:03:44.954560 18100 raft_consensus.cc:3037] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:03:44.956617 18234 raft_consensus.cc:2454] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8299f9a26ec44f97a65718498e2992f4 in term 1.
I20260620 15:03:44.956768 18100 raft_consensus.cc:2454] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8299f9a26ec44f97a65718498e2992f4 in term 1.
I20260620 15:03:44.957026 17899 leader_election.cc:302] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8299f9a26ec44f97a65718498e2992f4, 9a29005a18a84af99ad1f61aed3c0dfa; no voters:
I20260620 15:03:44.957306 18305 raft_consensus.cc:2781] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:03:44.957783 18305 raft_consensus.cc:686] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 1 LEADER]: Becoming Leader. State: Replica: 8299f9a26ec44f97a65718498e2992f4, State: Running, Role: LEADER
I20260620 15:03:44.958269 18305 consensus_queue.cc:237] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } }
I20260620 15:03:44.962853 17822 catalog_manager.cc:5495] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8299f9a26ec44f97a65718498e2992f4 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "8299f9a26ec44f97a65718498e2992f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } health_report { overall_health: HEALTHY } } }
W20260620 15:03:44.975867 18012 tablet.cc:2369] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:03:44.999056 18146 tablet.cc:2369] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:03:45.004980 18280 tablet.cc:2369] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:03:45.007490 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:45.001428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.2.70.1:40489)
0620 15:03:45.001651 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:45.001658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:45.001764 (+ 106us) server_negotiation.cc:407] Connection header received
0620 15:03:45.002016 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:45.002020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:45.002120 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:45.002275 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:03:45.002669 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:45.003404 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:45.004982 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:45.005597 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:45.005712 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:45.006263 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:03:45.006801 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:03:45.006924 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:45.007038 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 15:03:45.007114 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:03:45.016362 18234 raft_consensus.cc:1261] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 FOLLOWER]: Refusing update from remote peer 8299f9a26ec44f97a65718498e2992f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:45.016362 18100 raft_consensus.cc:1261] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 1 FOLLOWER]: Refusing update from remote peer 8299f9a26ec44f97a65718498e2992f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:03:45.017469 18305 consensus_queue.cc:1035] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:45.017810 18308 consensus_queue.cc:1035] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:03:45.028986 18311 mvcc.cc:205] Tried to move back new op lower bound from 7298940211256205312 to 7298940211032842240. Current Snapshot: MvccSnapshot[applied={T|T < 7298940211256205312}]
I20260620 15:03:45.042282 17822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:03:45.042589 17822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:03:45.046187 17822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Sending DeleteTablet for 3 replicas of tablet f7335b09d74347ac84fa38ffe30b8b92
I20260620 15:03:45.047385 18080 tablet_service.cc:1514] Processing DeleteTablet for tablet f7335b09d74347ac84fa38ffe30b8b92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:45 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:35692
I20260620 15:03:45.047360 18214 tablet_service.cc:1514] Processing DeleteTablet for tablet f7335b09d74347ac84fa38ffe30b8b92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:45 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:44936
I20260620 15:03:45.048296 18320 tablet_replica.cc:330] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: stopping tablet replica
I20260620 15:03:45.048285 17946 tablet_service.cc:1514] Processing DeleteTablet for tablet f7335b09d74347ac84fa38ffe30b8b92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:03:45 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:46948
I20260620 15:03:45.048456 18321 tablet_replica.cc:330] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: stopping tablet replica
I20260620 15:03:45.048768 18321 raft_consensus.cc:2229] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:03:45.048768 18320 raft_consensus.cc:2229] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:03:45.049247 18320 raft_consensus.cc:2258] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:45.049247 18321 raft_consensus.cc:2258] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:45.049685 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 17883
I20260620 15:03:45.050263 18322 tablet_replica.cc:330] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: stopping tablet replica
I20260620 15:03:45.050666 18322 raft_consensus.cc:2229] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:03:45.051314 18322 raft_consensus.cc:2258] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:03:45.051940 18321 ts_tablet_manager.cc:1905] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:45.052625 18320 ts_tablet_manager.cc:1905] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:45.058768 18321 ts_tablet_manager.cc:1918] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:45.059018 18321 log.cc:1196] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/f7335b09d74347ac84fa38ffe30b8b92
I20260620 15:03:45.059304 18320 ts_tablet_manager.cc:1918] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:45.059640 18321 ts_tablet_manager.cc:1939] T f7335b09d74347ac84fa38ffe30b8b92 P 9a29005a18a84af99ad1f61aed3c0dfa: Deleting consensus metadata
I20260620 15:03:45.061641 18322 ts_tablet_manager.cc:1905] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:03:45.061684 17806 catalog_manager.cc:4841] TS 9a29005a18a84af99ad1f61aed3c0dfa (127.2.70.3:34629): tablet f7335b09d74347ac84fa38ffe30b8b92 (table test-table [id=eb0a18cb5fe441c6bfa55fa2f41f3888]) successfully deleted
I20260620 15:03:45.063149 18320 log.cc:1196] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/f7335b09d74347ac84fa38ffe30b8b92
I20260620 15:03:45.063735 18320 ts_tablet_manager.cc:1939] T f7335b09d74347ac84fa38ffe30b8b92 P 5105d1414cb245b79dfbdf04d09cb863: Deleting consensus metadata
I20260620 15:03:45.065557 17809 catalog_manager.cc:4841] TS 5105d1414cb245b79dfbdf04d09cb863 (127.2.70.2:39563): tablet f7335b09d74347ac84fa38ffe30b8b92 (table test-table [id=eb0a18cb5fe441c6bfa55fa2f41f3888]) successfully deleted
I20260620 15:03:45.068400 18322 ts_tablet_manager.cc:1918] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:03:45.068584 18322 log.cc:1196] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/f7335b09d74347ac84fa38ffe30b8b92
I20260620 15:03:45.069173 18322 ts_tablet_manager.cc:1939] T f7335b09d74347ac84fa38ffe30b8b92 P 8299f9a26ec44f97a65718498e2992f4: Deleting consensus metadata
I20260620 15:03:45.071110 17809 catalog_manager.cc:4841] TS 8299f9a26ec44f97a65718498e2992f4 (127.2.70.1:40489): tablet f7335b09d74347ac84fa38ffe30b8b92 (table test-table [id=eb0a18cb5fe441c6bfa55fa2f41f3888]) successfully deleted
Jun 20 15:03:45 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:03:45.081208 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:45.050763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46966 (local address 127.2.70.1:40489)
0620 15:03:45.050954 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:45.050962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:45.060444 (+ 9482us) server_negotiation.cc:407] Connection header received
0620 15:03:45.060743 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:45.060749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:45.060833 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:45.060907 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:45.063062 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:45.064335 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:45.066820 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:45.067607 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:45.067745 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:45.073364 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:45.073375 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:45.073396 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:45.073437 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:45.076477 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:45.077319 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:45.077326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:45.077328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:45.077688 (+ 360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:45.079418 (+ 1730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:45.079426 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:45.079432 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:45.079799 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:45.080034 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:45.080572 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 15:03:45.080841 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:03:45.081965 18006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:45.181545 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 17883
I20260620 15:03:45.197044 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 18017
Jun 20 15:03:45 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:03:45.213325 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:45.197816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.2.70.2:39563)
0620 15:03:45.198026 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:45.198034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:45.198056 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:03:45.198290 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:45.198295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:45.198372 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:45.198477 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:45.199822 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:45.200544 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:45.202218 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:45.202610 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:45.202743 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:45.207017 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:45.207031 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:45.207048 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:45.207088 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:45.210464 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:45.211205 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:45.211213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:45.211216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:45.211353 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:45.211749 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:45.211754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:45.211757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:45.212152 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:45.212446 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:45.212761 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 15:03:45.212980 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:03:45.214087 18140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:45.304180 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 18017
I20260620 15:03:45.321424 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 18151
Jun 20 15:03:45 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:03:45.337174 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:03:45.322243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.2.70.3:34629)
0620 15:03:45.322433 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 15:03:45.322440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:03:45.322460 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:03:45.322679 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:03:45.322698 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:03:45.322800 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:03:45.322904 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:03:45.324374 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:45.325106 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:45.326486 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:03:45.326889 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:03:45.327015 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:03:45.331516 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:03:45.331533 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:03:45.331549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:03:45.331592 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:03:45.334449 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:45.335165 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:45.335172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:45.335174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:45.335303 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:03:45.335678 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:03:45.335684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:03:45.335687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:03:45.336029 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:03:45.336339 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0620 15:03:45.336621 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 15:03:45.336851 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:03:45.338011 18274 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:45.427505 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 18151
I20260620 15:03:45.445645 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 17790
I20260620 15:03:45.446781 17852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:03:45.563166 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 17790
I20260620 15:04:05.586424 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34899
--webserver_interface=127.2.70.62
--webserver_port=45599
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:05.927930 18328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:05.928215 18328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:05.928392 18328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:05.935757 18328 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:04:05.935856 18328 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:04:05.935896 18328 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:04:05.936002 18328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:04:05.936074 18328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:05.936220 18328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:05.936291 18328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:04:05.936378 18328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:04:05.946638 18328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=45599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:05.948010 18328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:05.949419 18328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:05.973996 18334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:05.974051 18335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:05.974084 18337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:05.974606 18328 server_base.cc:1034] running on GCE node
I20260620 15:04:05.975594 18328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:05.977288 18328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:05.978576 18328 hybrid_clock.cc:648] HybridClock initialized: now 1781967845978482 us; error 70 us; skew 500 ppm
I20260620 15:04:05.978986 18328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:05 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967845, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:05.987713 18328 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:04:05.989809 18328 webserver.cc:460] Webserver started at http://127.2.70.62:45599/ using document root <none> and password file <none>
I20260620 15:04:05.990682 18328 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:05.990846 18328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:05.997583 18328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 15:04:06.000902 18344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:06.002612 18328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:06.002835 18328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "bdce2c57c49e57bcf476c1e4cf923fc1"
server_key_iv: "24980c68bfdfec4560ae8f6086e2df23"
server_key_version: "encryptionkey@0"
I20260620 15:04:06.003985 18328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:06.018662 18328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:06.023878 18328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:06.024260 18328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:06.046073 18328 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:34899
I20260620 15:04:06.046106 18395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:34899 every 8 connection(s)
I20260620 15:04:06.047870 18328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:04:06.053335 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 18328
I20260620 15:04:06.054378 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:40489
--local_ip_for_outbound_sockets=127.2.70.1
--tserver_master_addrs=127.2.70.62:34899
--webserver_port=43285
--webserver_interface=127.2.70.1
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:04:06.056262 18396 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:04:06.061869 18396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Bootstrap starting.
I20260620 15:04:06.083034 18396 log.cc:824] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:06.095890 18396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:04:06.096511 18396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Bootstrap complete.
I20260620 15:04:06.106182 18396 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:04:06.107084 18396 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa95e4ac5f344d2aab0f7b63bc1d720e, State: Initialized, Role: FOLLOWER
I20260620 15:04:06.107918 18396 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:04:06.108263 18396 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:04:06.108387 18396 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:04:06.108621 18396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 1 FOLLOWER]: Advancing to term 2
I20260620 15:04:06.113024 18396 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:04:06.113685 18396 leader_election.cc:302] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa95e4ac5f344d2aab0f7b63bc1d720e; no voters:
I20260620 15:04:06.114367 18396 leader_election.cc:288] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 15:04:06.114519 18400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 15:04:06.115835 18400 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 2 LEADER]: Becoming Leader. State: Replica: fa95e4ac5f344d2aab0f7b63bc1d720e, State: Running, Role: LEADER
I20260620 15:04:06.116506 18400 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:04:06.116796 18396 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:04:06.119385 18402 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa95e4ac5f344d2aab0f7b63bc1d720e. Latest consensus state: current_term: 2 leader_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } } }
I20260620 15:04:06.119855 18402 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:06.120378 18401 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } } }
I20260620 15:04:06.120802 18401 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:06.121551 18406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:04:06.132306 18406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:04:06.133934 18406 catalog_manager.cc:1259] Loaded cluster ID: f340bedf6e9f4545a84d6d977ba35575
I20260620 15:04:06.134099 18406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:04:06.138586 18406 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:04:06.140255 18406 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Loaded TSK: 0
I20260620 15:04:06.173218 18406 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Generated new TSK 1
I20260620 15:04:06.173694 18406 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:04:06.416576 18398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:06.416903 18398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:06.417109 18398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:06.424816 18398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:06.424921 18398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:06.425088 18398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:04:06.435249 18398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:40489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=43285
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:06.436628 18398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:06.438071 18398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:06.453171 18424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:06.453372 18426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:06.453652 18423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:06.454244 18398 server_base.cc:1034] running on GCE node
I20260620 15:04:06.454905 18398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:06.456009 18398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:06.457249 18398 hybrid_clock.cc:648] HybridClock initialized: now 1781967846457188 us; error 55 us; skew 500 ppm
I20260620 15:04:06.457706 18398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:06 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:06.461038 18398 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:04:06.462658 18398 webserver.cc:460] Webserver started at http://127.2.70.1:43285/ using document root <none> and password file <none>
I20260620 15:04:06.463361 18398 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:06.463524 18398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:06.469194 18398 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 15:04:06.472249 18433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:06.473747 18398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:06.473918 18398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "8299f9a26ec44f97a65718498e2992f4"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "16901be973986c7a58999594d1eaf295"
server_key_iv: "a665a3bad7e76beb651e127657cc6453"
server_key_version: "encryptionkey@0"
I20260620 15:04:06.474917 18398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:06.495926 18398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:06.500325 18398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:06.500613 18398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:06.501698 18398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:06.503162 18398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:06.503295 18398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:06.503433 18398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:06.503523 18398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:06.543972 18398 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:40489
I20260620 15:04:06.544061 18545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:40489 every 8 connection(s)
I20260620 15:04:06.545429 18398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:04:06.552224 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 18398
I20260620 15:04:06.552892 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:39563
--local_ip_for_outbound_sockets=127.2.70.2
--tserver_master_addrs=127.2.70.62:34899
--webserver_port=38797
--webserver_interface=127.2.70.2
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:04:06 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:06.569940 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:06.548569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:38639)
0620 15:04:06.549472 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:06.549543 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:06.551293 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:06.552756 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:06.552770 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:06.553446 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:06.554035 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:06.554052 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:06.555633 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:06.555640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:06.556838 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:06.556850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:06.557947 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:06.557955 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:06.558236 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:06.558933 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:06.558962 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:06.561271 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:06.565866 (+ 4595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:06.565873 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:06.565884 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:06.566311 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:06.566794 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:06.566797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:06.566799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:06.566993 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:06.567958 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:06.567963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:06.568249 (+ 286us) client_negotiation.cc:769] Sending connection context
0620 15:04:06.568481 (+ 232us) client_negotiation.cc:240] Negotiation successful
0620 15:04:06.568846 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":389,"threads_started":1}
I20260620 15:04:06.570454 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:06.548298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:38639 (local address 127.2.70.62:34899)
0620 15:04:06.548870 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:06.548890 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:06.549673 (+ 783us) server_negotiation.cc:407] Connection header received
0620 15:04:06.551833 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:06.551846 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:06.552199 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:06.552578 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:06.554235 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:06.555476 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:06.556988 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:06.557707 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:06.557910 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:06.561589 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:06.561601 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:06.561623 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:06.561684 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:06.565730 (+ 4046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:06.566456 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:06.566467 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:06.566477 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:06.566683 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:06.567112 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:06.567120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:06.567126 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:06.567813 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:06.568003 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:06.568725 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 15:04:06.569167 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":246,"threads_started":1}
I20260620 15:04:06.573251 18546 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34899
I20260620 15:04:06.573622 18546 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:06.574564 18546 heartbeater.cc:511] Master 127.2.70.62:34899 requested a full tablet report, sending...
I20260620 15:04:06.576879 18361 ts_manager.cc:194] Registered new tserver with Master: 8299f9a26ec44f97a65718498e2992f4 (127.2.70.1:40489)
I20260620 15:04:06.578508 18361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:38639
W20260620 15:04:06.888301 18550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:06.888666 18550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:06.888854 18550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:06.896188 18550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:06.896301 18550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:06.896494 18550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:04:06.906428 18550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:39563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=38797
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:06.907943 18550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:06.909386 18550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:06.923977 18555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:06.923951 18556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:06.924193 18558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:06.925027 18550 server_base.cc:1034] running on GCE node
I20260620 15:04:06.925783 18550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:06.926859 18550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:06.928117 18550 hybrid_clock.cc:648] HybridClock initialized: now 1781967846928025 us; error 70 us; skew 500 ppm
I20260620 15:04:06.928514 18550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:06 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:06.932016 18550 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:04:06.933764 18550 webserver.cc:460] Webserver started at http://127.2.70.2:38797/ using document root <none> and password file <none>
I20260620 15:04:06.934526 18550 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:06.934653 18550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:06.942588 18550 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.007s
I20260620 15:04:06.946571 18565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:06.948189 18550 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260620 15:04:06.948374 18550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "5105d1414cb245b79dfbdf04d09cb863"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "982282d0e55b287d7f58bfa2816b20cb"
server_key_iv: "811f3c78c14f83464710b70f1fd7b13d"
server_key_version: "encryptionkey@0"
I20260620 15:04:06.949433 18550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:06.966039 18550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:06.970499 18550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:06.970779 18550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:06.971853 18550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:06.973258 18550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:06.973345 18550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:06.973460 18550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:06.973587 18550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260620 15:04:07.007387 18550 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:39563
I20260620 15:04:07.007489 18677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:39563 every 8 connection(s)
I20260620 15:04:07.008942 18550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:04:07.009406 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 18550
I20260620 15:04:07.010203 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:34629
--local_ip_for_outbound_sockets=127.2.70.3
--tserver_master_addrs=127.2.70.62:34899
--webserver_port=39259
--webserver_interface=127.2.70.3
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 15:04:07 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967846, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:07.035014 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.011791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:46319 (local address 127.2.70.62:34899)
0620 15:04:07.012022 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.012030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.013307 (+ 1277us) server_negotiation.cc:407] Connection header received
0620 15:04:07.016288 (+ 2981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.016293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.016369 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.016471 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:07.018723 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.019417 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.021030 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.021464 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.021651 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.026038 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:07.026044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:07.026048 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:07.026091 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:07.030446 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.031615 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.031624 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.031630 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.031818 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.032496 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.032505 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.032510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.032973 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:07.033156 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.034234 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.034541 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:04:07.035987 18680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.012257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:46319)
0620 15:04:07.013110 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:07.013166 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:07.015965 (+ 2799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:07.016755 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:07.016770 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:07.017664 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:07.018530 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.018553 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.019587 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.019593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.020869 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.020880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.021749 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.021756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.022071 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.022818 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:07.022854 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:07.025732 (+ 2878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:07.030810 (+ 5078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:07.030819 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:07.030836 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:07.031438 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:07.031946 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:07.031950 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:07.031952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:07.032346 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:07.033112 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:07.033119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:07.033785 (+ 666us) client_negotiation.cc:769] Sending connection context
0620 15:04:07.034069 (+ 284us) client_negotiation.cc:240] Negotiation successful
0620 15:04:07.034541 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":321,"threads_started":1}
I20260620 15:04:07.037956 18678 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34899
I20260620 15:04:07.038311 18678 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:07.039070 18678 heartbeater.cc:511] Master 127.2.70.62:34899 requested a full tablet report, sending...
I20260620 15:04:07.040551 18361 ts_manager.cc:194] Registered new tserver with Master: 5105d1414cb245b79dfbdf04d09cb863 (127.2.70.2:39563)
I20260620 15:04:07.041368 18361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:46319
W20260620 15:04:07.343914 18681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:07.344190 18681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:07.344359 18681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:07.351585 18681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:07.351722 18681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:07.351918 18681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:04:07.361979 18681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:34629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=39259
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:07.363296 18681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:07.364626 18681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:07.377575 18686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:07.377625 18689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:07.377540 18687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:07.378479 18681 server_base.cc:1034] running on GCE node
I20260620 15:04:07.379122 18681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:07.380076 18681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:07.381311 18681 hybrid_clock.cc:648] HybridClock initialized: now 1781967847381253 us; error 67 us; skew 500 ppm
I20260620 15:04:07.381697 18681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:07 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967847, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:07.384853 18681 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:04:07.386363 18681 webserver.cc:460] Webserver started at http://127.2.70.3:39259/ using document root <none> and password file <none>
I20260620 15:04:07.387091 18681 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:07.387250 18681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:07.392881 18681 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:04:07.396121 18696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:07.397941 18681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:07.398150 18681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "9a29005a18a84af99ad1f61aed3c0dfa"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "0e8b5de7671595e8650a31106e30497b"
server_key_iv: "de81e59b6b90abca84a6fc7616cf79c4"
server_key_version: "encryptionkey@0"
I20260620 15:04:07.399219 18681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:07.424898 18681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:07.429388 18681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:07.429699 18681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:07.430728 18681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:07.432147 18681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:07.432276 18681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:07.432422 18681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:07.432512 18681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:07.467578 18681 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:34629
I20260620 15:04:07.467634 18808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:34629 every 8 connection(s)
I20260620 15:04:07.469045 18681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:04:07.477780 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 18681
Jun 20 15:04:07 dist-test-slave-n5vv krb5kdc[17777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967847, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:07.493454 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.472024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:33755 (local address 127.2.70.62:34899)
0620 15:04:07.472252 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.472258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.473696 (+ 1438us) server_negotiation.cc:407] Connection header received
0620 15:04:07.475838 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.475843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.475917 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.476017 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:07.478090 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.479499 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.481211 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.481806 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.482016 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.486024 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:07.486032 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:07.486037 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:07.486079 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:07.489770 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.490706 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.490711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.490715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.490872 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.491362 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.491367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.491371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.491766 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:07.491944 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.492782 (+ 838us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.493037 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260620 15:04:07.494541 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.472523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:33755)
0620 15:04:07.473418 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:07.473470 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:07.475508 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:07.476330 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:07.476340 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:07.476990 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:07.477827 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.477884 (+ 57us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.479671 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.479683 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.481037 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.481050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.482109 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.482118 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.482473 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.483225 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:07.483249 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:07.485780 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:07.489942 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:07.489953 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:07.489977 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:07.490536 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:07.491017 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:07.491020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:07.491023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:07.491220 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:07.491927 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:07.491934 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:07.492282 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 15:04:07.492658 (+ 376us) client_negotiation.cc:240] Negotiation successful
0620 15:04:07.493241 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":355,"threads_started":1}
I20260620 15:04:07.494891 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.478793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37102 (local address 127.2.70.62:34899)
0620 15:04:07.479472 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.479478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.479712 (+ 234us) server_negotiation.cc:407] Connection header received
0620 15:04:07.479929 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.479940 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.479996 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.480110 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:07.481761 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.482642 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.483995 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.484518 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.484648 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.488921 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:07.488926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:07.488929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:07.488978 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:07.492078 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.492738 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.492744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.492747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.492865 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.493255 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.493261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.493264 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.493644 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:07.493879 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.494263 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.494481 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":318,"threads_started":1}
I20260620 15:04:07.496412 18809 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34899
I20260620 15:04:07.496768 18809 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:07.497699 18809 heartbeater.cc:511] Master 127.2.70.62:34899 requested a full tablet report, sending...
I20260620 15:04:07.499140 18360 ts_manager.cc:194] Registered new tserver with Master: 9a29005a18a84af99ad1f61aed3c0dfa (127.2.70.3:34629)
I20260620 15:04:07.500052 18360 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:33755
I20260620 15:04:07.509837 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:04:07.517020 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.510711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.2.70.62:34899)
0620 15:04:07.510912 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.510919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.511419 (+ 500us) server_negotiation.cc:407] Connection header received
0620 15:04:07.511586 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.511590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.511725 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.511846 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:04:07.512345 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.513095 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.514969 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.515395 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.515521 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.516117 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:04:07.516339 (+ 222us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 15:04:07.516713 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37108: token expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:04:07.517177 18812 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37108: token expired
W20260620 15:04:07.517812 18283 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.70.62:34899)
I20260620 15:04:07.531291 18812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.518352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.2.70.62:34899)
0620 15:04:07.518519 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.518525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.518544 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:04:07.518697 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.518700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.518759 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.518836 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:07.520343 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.521044 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.522745 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.523166 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.523257 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.526012 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:07.526016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:07.526019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:07.526050 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:07.528607 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.529316 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.529319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.529322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.529454 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.529928 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.529932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.529934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.530234 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:07.530413 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.530763 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.530940 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:04:07.534698 18360 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37118:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:04:07.541857 18360 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:04:07.567636 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.555614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34650)
0620 15:04:07.556548 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:07.556599 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:07.556817 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:07.559384 (+ 2567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:07.559393 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:07.559422 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:07.559964 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.559980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.561708 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.561716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.564654 (+ 2938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.564673 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.566522 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.566528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.566752 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.566837 (+ 85us) client_negotiation.cc:769] Sending connection context
0620 15:04:07.566979 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 15:04:07.567172 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":1,"thread_start_us":403,"threads_started":1}
I20260620 15:04:07.568578 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.555882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34650 (local address 127.2.70.1:40489)
0620 15:04:07.557776 (+ 1894us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.557785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.557817 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:04:07.557958 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.557972 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.558333 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.558598 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:07.560324 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.561438 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.564841 (+ 3403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.566259 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.566416 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.566671 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.568054 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.568241 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1691,"thread_start_us":1483,"threads_started":1}
I20260620 15:04:07.568955 18821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.556902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:39563 (local address 127.0.0.1:44320)
0620 15:04:07.557992 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:07.558018 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:07.558129 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:07.559944 (+ 1815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:07.559951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:07.559985 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:07.560434 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.560450 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.562802 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.562811 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.565853 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.565869 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.568175 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.568183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.568356 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.568390 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:04:07.568487 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 15:04:07.568623 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"thread_start_us":851,"threads_started":1}
I20260620 15:04:07.569286 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.557447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44320 (local address 127.2.70.2:39563)
0620 15:04:07.559217 (+ 1770us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.559228 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.559262 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:04:07.559386 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.559397 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.559680 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.560681 (+ 1001us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:07.560956 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.562656 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.566035 (+ 3379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.567681 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.568060 (+ 379us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.568286 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.568559 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.568839 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1579,"thread_start_us":1365,"threads_started":1}
I20260620 15:04:07.571676 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.558113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:34629 (local address 127.0.0.1:45936)
0620 15:04:07.562055 (+ 3942us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:07.562074 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:07.562206 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:07.563229 (+ 1023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:07.563236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:07.563298 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:07.563902 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.563924 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.565695 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.565709 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.568167 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.568182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.570784 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.570791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.570983 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.571029 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:04:07.571154 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 15:04:07.571291 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3752,"thread_start_us":3586,"threads_started":1}
I20260620 15:04:07.572613 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.557893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.2.70.3:34629)
0620 15:04:07.558721 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.558733 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.562183 (+ 3450us) server_negotiation.cc:407] Connection header received
0620 15:04:07.562348 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.562355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.562566 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.562757 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:07.564133 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.565560 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.568364 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.570556 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.570722 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.570982 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.571208 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.572315 (+ 1107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":497,"threads_started":1}
I20260620 15:04:07.572748 18613 tablet_service.cc:1467] Processing CreateTablet for tablet b36c8807e7ac4539ae61c1f717ee3130 (DEFAULT_TABLE table=test-table [id=bc3b02a30ec147d98d7fe1ac4a60dd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:07.572602 18481 tablet_service.cc:1467] Processing CreateTablet for tablet b36c8807e7ac4539ae61c1f717ee3130 (DEFAULT_TABLE table=test-table [id=bc3b02a30ec147d98d7fe1ac4a60dd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:07.574129 18481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36c8807e7ac4539ae61c1f717ee3130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:07.574127 18613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36c8807e7ac4539ae61c1f717ee3130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:07.575562 18744 tablet_service.cc:1467] Processing CreateTablet for tablet b36c8807e7ac4539ae61c1f717ee3130 (DEFAULT_TABLE table=test-table [id=bc3b02a30ec147d98d7fe1ac4a60dd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:07.576756 18744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b36c8807e7ac4539ae61c1f717ee3130. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:07.582266 18546 heartbeater.cc:503] Master 127.2.70.62:34899 was elected leader, sending a full tablet report...
I20260620 15:04:07.592219 18826 tablet_bootstrap.cc:492] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Bootstrap starting.
I20260620 15:04:07.594234 18825 tablet_bootstrap.cc:492] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Bootstrap starting.
I20260620 15:04:07.595678 18827 tablet_bootstrap.cc:492] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Bootstrap starting.
I20260620 15:04:07.598330 18826 tablet_bootstrap.cc:654] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:07.598585 18825 tablet_bootstrap.cc:654] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:07.599262 18827 tablet_bootstrap.cc:654] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:07.599843 18825 log.cc:824] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:07.599859 18826 log.cc:824] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:07.600437 18827 log.cc:824] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:07.603749 18827 tablet_bootstrap.cc:492] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: No bootstrap required, opened a new log
I20260620 15:04:07.603911 18826 tablet_bootstrap.cc:492] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: No bootstrap required, opened a new log
I20260620 15:04:07.603911 18825 tablet_bootstrap.cc:492] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: No bootstrap required, opened a new log
I20260620 15:04:07.604094 18827 ts_tablet_manager.cc:1397] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260620 15:04:07.604188 18826 ts_tablet_manager.cc:1397] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260620 15:04:07.604188 18825 ts_tablet_manager.cc:1397] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260620 15:04:07.613389 18827 raft_consensus.cc:348] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.613920 18827 raft_consensus.cc:374] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:07.614037 18827 raft_consensus.cc:729] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a29005a18a84af99ad1f61aed3c0dfa, State: Initialized, Role: FOLLOWER
I20260620 15:04:07.614770 18827 consensus_queue.cc:260] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.616331 18825 raft_consensus.cc:348] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.616343 18826 raft_consensus.cc:348] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.616807 18826 raft_consensus.cc:374] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:07.616825 18827 ts_tablet_manager.cc:1428] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260620 15:04:07.616807 18825 raft_consensus.cc:374] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:07.616909 18826 raft_consensus.cc:729] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8299f9a26ec44f97a65718498e2992f4, State: Initialized, Role: FOLLOWER
I20260620 15:04:07.616954 18825 raft_consensus.cc:729] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5105d1414cb245b79dfbdf04d09cb863, State: Initialized, Role: FOLLOWER
I20260620 15:04:07.617352 18809 heartbeater.cc:503] Master 127.2.70.62:34899 was elected leader, sending a full tablet report...
I20260620 15:04:07.617748 18826 consensus_queue.cc:260] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.617748 18825 consensus_queue.cc:260] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.619521 18678 heartbeater.cc:503] Master 127.2.70.62:34899 was elected leader, sending a full tablet report...
I20260620 15:04:07.619894 18826 ts_tablet_manager.cc:1428] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260620 15:04:07.621521 18825 ts_tablet_manager.cc:1428] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260620 15:04:07.720834 18810 tablet.cc:2369] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:04:07.761400 18679 tablet.cc:2369] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:04:07.797941 18547 tablet.cc:2369] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:04:07.812757 18831 raft_consensus.cc:482] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:04:07.813124 18831 raft_consensus.cc:504] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.814879 18831 leader_election.cc:288] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8299f9a26ec44f97a65718498e2992f4 (127.2.70.1:40489), 5105d1414cb245b79dfbdf04d09cb863 (127.2.70.2:39563)
I20260620 15:04:07.822198 18811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.815296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.2.70.3:35879)
0620 15:04:07.815556 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:07.815574 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:07.815682 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:07.816309 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:07.816315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:07.816349 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:07.816613 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.816625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.818029 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.818033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.819968 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.819983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.821388 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.821393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.821629 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.821655 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 15:04:07.821735 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:04:07.821828 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":30}
I20260620 15:04:07.822184 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.815407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:35879 (local address 127.2.70.1:40489)
0620 15:04:07.815646 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.815654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.815679 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:04:07.815948 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.815953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.816040 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.816152 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:07.816825 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.817873 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.820143 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.821209 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.821372 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.821510 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.821792 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.821934 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 15:04:07.822863 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.815977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:37615 (local address 127.2.70.2:39563)
0620 15:04:07.816193 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.816202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.816606 (+ 404us) server_negotiation.cc:407] Connection header received
0620 15:04:07.816839 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.816844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.816941 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.817063 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:07.817590 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.818418 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.820596 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.821872 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.822000 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.822090 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.822538 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.822633 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:04:07.823645 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.815823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:39563 (local address 127.2.70.3:37615)
0620 15:04:07.816496 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:07.816512 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:07.816630 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:07.817063 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:07.817066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:07.817085 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:07.817398 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.817408 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.818544 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.818549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.820442 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:07.820452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.822150 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.822158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:07.822337 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.822372 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:04:07.823101 (+ 729us) client_negotiation.cc:240] Negotiation successful
0620 15:04:07.823246 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":1,"thread_start_us":390,"threads_started":1}
I20260620 15:04:07.823423 18501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36c8807e7ac4539ae61c1f717ee3130" candidate_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8299f9a26ec44f97a65718498e2992f4" is_pre_election: true
I20260620 15:04:07.824055 18501 raft_consensus.cc:2454] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a29005a18a84af99ad1f61aed3c0dfa in term 0.
I20260620 15:04:07.824544 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36c8807e7ac4539ae61c1f717ee3130" candidate_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5105d1414cb245b79dfbdf04d09cb863" is_pre_election: true
I20260620 15:04:07.824741 18700 leader_election.cc:302] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8299f9a26ec44f97a65718498e2992f4, 9a29005a18a84af99ad1f61aed3c0dfa; no voters:
I20260620 15:04:07.825069 18633 raft_consensus.cc:2454] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a29005a18a84af99ad1f61aed3c0dfa in term 0.
I20260620 15:04:07.825184 18831 raft_consensus.cc:2781] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:04:07.825335 18831 raft_consensus.cc:482] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:04:07.825436 18831 raft_consensus.cc:3037] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:07.827666 18831 raft_consensus.cc:504] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.828289 18831 leader_election.cc:288] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [CANDIDATE]: Term 1 election: Requested vote from peers 8299f9a26ec44f97a65718498e2992f4 (127.2.70.1:40489), 5105d1414cb245b79dfbdf04d09cb863 (127.2.70.2:39563)
I20260620 15:04:07.828801 18501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36c8807e7ac4539ae61c1f717ee3130" candidate_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8299f9a26ec44f97a65718498e2992f4"
I20260620 15:04:07.829054 18501 raft_consensus.cc:3037] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:07.829030 18633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b36c8807e7ac4539ae61c1f717ee3130" candidate_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5105d1414cb245b79dfbdf04d09cb863"
I20260620 15:04:07.829247 18633 raft_consensus.cc:3037] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:07.832434 18633 raft_consensus.cc:2454] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a29005a18a84af99ad1f61aed3c0dfa in term 1.
I20260620 15:04:07.832442 18501 raft_consensus.cc:2454] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a29005a18a84af99ad1f61aed3c0dfa in term 1.
I20260620 15:04:07.832870 18700 leader_election.cc:302] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5105d1414cb245b79dfbdf04d09cb863, 9a29005a18a84af99ad1f61aed3c0dfa; no voters:
I20260620 15:04:07.833175 18831 raft_consensus.cc:2781] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:04:07.833676 18831 raft_consensus.cc:686] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 LEADER]: Becoming Leader. State: Replica: 9a29005a18a84af99ad1f61aed3c0dfa, State: Running, Role: LEADER
I20260620 15:04:07.834280 18831 consensus_queue.cc:237] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } }
I20260620 15:04:07.839323 18361 catalog_manager.cc:5495] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a29005a18a84af99ad1f61aed3c0dfa (127.2.70.3). New cstate: current_term: 1 leader_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a29005a18a84af99ad1f61aed3c0dfa" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 34629 } health_report { overall_health: HEALTHY } } }
I20260620 15:04:07.875066 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.869375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45940 (local address 127.2.70.3:34629)
0620 15:04:07.869630 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.869637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.869750 (+ 113us) server_negotiation.cc:407] Connection header received
0620 15:04:07.870030 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.870035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.870124 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.870296 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:04:07.870684 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.871473 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.872782 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.873206 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.873364 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.873939 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:04:07.874477 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:04:07.874651 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.874705 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.874792 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 15:04:07.884194 18501 raft_consensus.cc:1261] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 1 FOLLOWER]: Refusing update from remote peer 9a29005a18a84af99ad1f61aed3c0dfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:04:07.884197 18633 raft_consensus.cc:1261] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 1 FOLLOWER]: Refusing update from remote peer 9a29005a18a84af99ad1f61aed3c0dfa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:04:07.885206 18831 consensus_queue.cc:1035] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5105d1414cb245b79dfbdf04d09cb863" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:04:07.885543 18835 consensus_queue.cc:1035] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "8299f9a26ec44f97a65718498e2992f4" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:04:07.901312 18840 mvcc.cc:205] Tried to move back new op lower bound from 7298940304922435584 to 7298940304733204480. Current Snapshot: MvccSnapshot[applied={T|T < 7298940304922435584}]
I20260620 15:04:07.913340 18361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:04:07.913679 18361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37118:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:04:07.917307 18361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Sending DeleteTablet for 3 replicas of tablet b36c8807e7ac4539ae61c1f717ee3130
I20260620 15:04:07.918829 18481 tablet_service.cc:1514] Processing DeleteTablet for tablet b36c8807e7ac4539ae61c1f717ee3130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:07 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:34650
I20260620 15:04:07.918987 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 18398
I20260620 15:04:07.919174 18744 tablet_service.cc:1514] Processing DeleteTablet for tablet b36c8807e7ac4539ae61c1f717ee3130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:07 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:45936
I20260620 15:04:07.919186 18613 tablet_service.cc:1514] Processing DeleteTablet for tablet b36c8807e7ac4539ae61c1f717ee3130 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:07 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:44320
I20260620 15:04:07.921533 18849 tablet_replica.cc:330] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: stopping tablet replica
I20260620 15:04:07.922166 18849 raft_consensus.cc:2229] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:04:07.922894 18849 raft_consensus.cc:2258] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:07.923913 18848 tablet_replica.cc:330] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: stopping tablet replica
I20260620 15:04:07.924109 18847 tablet_replica.cc:330] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: stopping tablet replica
I20260620 15:04:07.924566 18847 raft_consensus.cc:2229] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:04:07.925103 18847 raft_consensus.cc:2258] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:07.925966 18848 raft_consensus.cc:2229] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:04:07.926159 18849 ts_tablet_manager.cc:1905] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:07.926702 18848 raft_consensus.cc:2258] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:07.928741 18848 ts_tablet_manager.cc:1905] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:07.933451 18849 ts_tablet_manager.cc:1918] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:07.933692 18849 log.cc:1196] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/b36c8807e7ac4539ae61c1f717ee3130
I20260620 15:04:07.934286 18849 ts_tablet_manager.cc:1939] T b36c8807e7ac4539ae61c1f717ee3130 P 5105d1414cb245b79dfbdf04d09cb863: Deleting consensus metadata
I20260620 15:04:07.936033 18848 ts_tablet_manager.cc:1918] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:07.936245 18848 log.cc:1196] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/b36c8807e7ac4539ae61c1f717ee3130
I20260620 15:04:07.936301 18348 catalog_manager.cc:4841] TS 5105d1414cb245b79dfbdf04d09cb863 (127.2.70.2:39563): tablet b36c8807e7ac4539ae61c1f717ee3130 (table test-table [id=bc3b02a30ec147d98d7fe1ac4a60dd49]) successfully deleted
I20260620 15:04:07.936821 18848 ts_tablet_manager.cc:1939] T b36c8807e7ac4539ae61c1f717ee3130 P 9a29005a18a84af99ad1f61aed3c0dfa: Deleting consensus metadata
I20260620 15:04:07.938519 18847 ts_tablet_manager.cc:1905] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:07.938683 18345 catalog_manager.cc:4841] TS 9a29005a18a84af99ad1f61aed3c0dfa (127.2.70.3:34629): tablet b36c8807e7ac4539ae61c1f717ee3130 (table test-table [id=bc3b02a30ec147d98d7fe1ac4a60dd49]) successfully deleted
I20260620 15:04:07.939568 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:07.919887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.2.70.1:40489)
0620 15:04:07.920114 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:07.920123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:07.920149 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:04:07.920367 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:07.920377 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:07.920537 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:07.920676 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:07.922690 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.923924 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.926478 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:07.927155 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:07.927285 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:07.931086 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:07.931102 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:07.931121 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:07.931168 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:07.935439 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.936436 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.936446 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.936450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.936619 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:07.937095 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:07.937103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:07.937110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:07.937614 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:07.937942 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:07.939091 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0620 15:04:07.939265 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:04:07.940987 18541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:08.050009 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 18398
W20260620 15:04:08.075314 18348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.1:40489 (error 108)
I20260620 15:04:08.075600 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 18550
I20260620 15:04:08.077728 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.076868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34664)
0620 15:04:08.077333 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:08.077509 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":228,"threads_started":1}
W20260620 15:04:08.078032 18348 catalog_manager.cc:4584] TS 8299f9a26ec44f97a65718498e2992f4 (127.2.70.1:40489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b36c8807e7ac4539ae61c1f717ee3130: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
I20260620 15:04:08.092319 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.076973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44336 (local address 127.2.70.2:39563)
0620 15:04:08.077204 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:08.077209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:08.077228 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:04:08.077297 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:08.077302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:08.077411 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:08.077547 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:08.079317 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:08.080088 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:08.082251 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:08.082664 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:08.082762 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:08.086244 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:08.086258 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:08.086274 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:08.086308 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:08.089406 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:08.090191 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:08.090205 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:08.090207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:08.090333 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:08.090728 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:08.090734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:08.090737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:08.091128 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:08.091448 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:08.091757 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 15:04:08.091982 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 15:04:08.093152 18673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:08.131860 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.131312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34680)
0620 15:04:08.131565 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:08.131685 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 15:04:08.133121 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.132664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34694)
0620 15:04:08.132865 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:08.132957 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:04:08.185550 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 18550
I20260620 15:04:08.193954 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.193276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34700)
0620 15:04:08.193608 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:08.193741 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260620 15:04:08.195335 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.194851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34712)
0620 15:04:08.195041 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:08.195149 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 15:04:08.205327 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 18681
I20260620 15:04:08.221899 18823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.206287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.2.70.3:34629)
0620 15:04:08.206497 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:08.206503 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:08.206526 (+ 23us) server_negotiation.cc:407] Connection header received
0620 15:04:08.206672 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:08.206679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:08.206768 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:08.206898 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:08.208308 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:08.209779 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:08.211131 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:08.211840 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:08.211983 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:08.214530 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:08.214546 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:08.214569 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:08.214618 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:08.218814 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:08.219537 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:08.219547 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:08.219551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:08.219725 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:08.220082 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:08.220087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:08.220090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:08.220599 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:08.220923 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:08.221219 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 15:04:08.221471 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:04:08.222819 18804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:08.275779 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.275179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34726)
0620 15:04:08.275457 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:08.275575 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 15:04:08.277029 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.276553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34730)
0620 15:04:08.276774 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:08.276858 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 15:04:08.348047 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 18681
I20260620 15:04:08.367712 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 18328
I20260620 15:04:08.369014 18391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:08.486819 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 18328
I20260620 15:04:08.486877 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:08.486292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40489 (local address 127.0.0.1:34732)
0620 15:04:08.486545 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:08.486692 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 15:04:28.510962 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34899
--webserver_interface=127.2.70.62
--webserver_port=45599
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:28.843096 18856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:28.843379 18856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:28.843469 18856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:28.851424 18856 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 15:04:28.851560 18856 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 15:04:28.851687 18856 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 15:04:28.851768 18856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:04:28.851848 18856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:28.851971 18856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:28.852044 18856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:04:28.852125 18856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:04:28.863399 18856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=45599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:28.864737 18856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:28.866142 18856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:28.878551 18861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:28.878638 18864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:28.878620 18862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:28.884662 18856 server_base.cc:1034] running on GCE node
I20260620 15:04:28.885473 18856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:28.887019 18856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:28.888293 18856 hybrid_clock.cc:648] HybridClock initialized: now 1781967868888221 us; error 46 us; skew 500 ppm
I20260620 15:04:28.888644 18856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:28 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967868, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:28.891984 18856 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:04:28.893610 18856 webserver.cc:460] Webserver started at http://127.2.70.62:45599/ using document root <none> and password file <none>
I20260620 15:04:28.894403 18856 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:28.894526 18856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:28.900166 18856 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 15:04:28.903486 18871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:28.904764 18856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:04:28.904943 18856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "bdce2c57c49e57bcf476c1e4cf923fc1"
server_key_iv: "24980c68bfdfec4560ae8f6086e2df23"
server_key_version: "encryptionkey@0"
I20260620 15:04:28.906026 18856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:28.922576 18856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:28.926489 18856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:28.926782 18856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:28.945933 18856 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:34899
I20260620 15:04:28.945979 18922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:34899 every 8 connection(s)
I20260620 15:04:28.947455 18856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:04:28.954417 18923 sys_catalog.cc:263] Verifying existing consensus state
I20260620 15:04:28.955663 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 18856
I20260620 15:04:28.956701 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:40489
--local_ip_for_outbound_sockets=127.2.70.1
--tserver_master_addrs=127.2.70.62:34899
--webserver_port=43285
--webserver_interface=127.2.70.1
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:04:28.959034 18923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Bootstrap starting.
I20260620 15:04:28.981370 18923 log.cc:824] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:28.997727 18923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 15:04:28.998512 18923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Bootstrap complete.
I20260620 15:04:29.007874 18923 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:04:29.008749 18923 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa95e4ac5f344d2aab0f7b63bc1d720e, State: Initialized, Role: FOLLOWER
I20260620 15:04:29.009470 18923 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:04:29.009771 18923 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:04:29.009893 18923 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:04:29.010085 18923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 2 FOLLOWER]: Advancing to term 3
I20260620 15:04:29.014379 18923 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:04:29.014990 18923 leader_election.cc:302] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa95e4ac5f344d2aab0f7b63bc1d720e; no voters:
I20260620 15:04:29.015722 18923 leader_election.cc:288] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 15:04:29.015878 18927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 15:04:29.017026 18927 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [term 3 LEADER]: Becoming Leader. State: Replica: fa95e4ac5f344d2aab0f7b63bc1d720e, State: Running, Role: LEADER
I20260620 15:04:29.017660 18927 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } }
I20260620 15:04:29.018092 18923 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:04:29.020118 18929 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa95e4ac5f344d2aab0f7b63bc1d720e. Latest consensus state: current_term: 3 leader_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } } }
I20260620 15:04:29.020491 18929 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:29.021040 18928 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa95e4ac5f344d2aab0f7b63bc1d720e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34899 } } }
I20260620 15:04:29.021359 18928 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:29.031172 18933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:04:29.041728 18933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:04:29.043663 18933 catalog_manager.cc:1259] Loaded cluster ID: f340bedf6e9f4545a84d6d977ba35575
I20260620 15:04:29.043794 18933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:04:29.049127 18933 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:04:29.051362 18933 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Loaded TSK: 0
I20260620 15:04:29.051543 18933 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fa95e4ac5f344d2aab0f7b63bc1d720e: Loaded TSK: 1
F20260620 15:04:29.053068 18933 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f24e8e3f9ad google::LogMessage::Fail() at ??:0
@ 0x7f24e8e4288a google::LogMessage::SendToLog() at ??:0
@ 0x7f24e8e3f4de google::LogMessage::Flush() at ??:0
@ 0x7f24e8e40a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f24ecba37d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f24ecb9273e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f24fe86e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f24fe86d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f24fe8f92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f24fe8f90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f24fe9435e9 std::function<>::operator()() at ??:0
@ 0x7f24fe871cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f24fe86fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f24fe8f7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f24fe8f7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f24eba8b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f24eba8e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f24eba8e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f24eba69eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f24eed436db start_thread at ??:0
@ 0x7f24e7cd961f clone at ??:0
W20260620 15:04:29.360924 18925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:29.361209 18925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:29.361389 18925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:29.368842 18925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:29.368961 18925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:29.369149 18925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:04:29.379261 18925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:40489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=43285
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:29.380674 18925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:29.382073 18925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:29.395594 18971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:29.395728 18970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:29.395680 18973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:29.396497 18925 server_base.cc:1034] running on GCE node
I20260620 15:04:29.397140 18925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:29.398197 18925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:29.399436 18925 hybrid_clock.cc:648] HybridClock initialized: now 1781967869399388 us; error 72 us; skew 500 ppm
I20260620 15:04:29.399756 18925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:29 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:29.403077 18925 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:04:29.404677 18925 webserver.cc:460] Webserver started at http://127.2.70.1:43285/ using document root <none> and password file <none>
I20260620 15:04:29.405369 18925 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:29.405555 18925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:29.411041 18925 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 15:04:29.414301 18980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:29.415843 18925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:04:29.416069 18925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "8299f9a26ec44f97a65718498e2992f4"
format_stamp: "Formatted at 2026-06-20 15:03:43 on dist-test-slave-n5vv"
server_key: "16901be973986c7a58999594d1eaf295"
server_key_iv: "a665a3bad7e76beb651e127657cc6453"
server_key_version: "encryptionkey@0"
I20260620 15:04:29.417119 18925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:29.452539 18925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:29.457370 18925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:29.457751 18925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:29.458958 18925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:29.461022 18987 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260620 15:04:29.467186 18987 ts_tablet_manager.cc:1852] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 15:04:29.468290 18987 ts_tablet_manager.cc:1905] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:29.476230 18987 ts_tablet_manager.cc:1918] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:29.476469 18987 log.cc:1196] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/b36c8807e7ac4539ae61c1f717ee3130
I20260620 15:04:29.477042 18987 ts_tablet_manager.cc:1939] T b36c8807e7ac4539ae61c1f717ee3130 P 8299f9a26ec44f97a65718498e2992f4: Deleting consensus metadata
I20260620 15:04:29.477526 18925 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 15:04:29.477666 18925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.001s sys 0.000s
I20260620 15:04:29.477807 18925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:29.477910 18925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:29.514438 18925 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:40489
I20260620 15:04:29.514487 19093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:40489 every 8 connection(s)
I20260620 15:04:29.515884 18925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:04:29.521176 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:29.519144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:42409)
0620 15:04:29.519913 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:29.520122 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":286,"threads_started":1}
W20260620 15:04:29.521981 19094 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:34899 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.62:34899: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
I20260620 15:04:29.522804 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:29.522498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:53953)
0620 15:04:29.522635 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:29.522702 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 15:04:29.523617 19096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:29.523306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:57181)
0620 15:04:29.523432 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:29.523503 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:04:29.523845 19094 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:04:29.525625 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 18925
I20260620 15:04:29.526221 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:39563
--local_ip_for_outbound_sockets=127.2.70.2
--tserver_master_addrs=127.2.70.62:34899
--webserver_port=38797
--webserver_interface=127.2.70.2
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:29.861334 19097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:29.861712 19097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:29.861831 19097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:29.869226 19097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:29.869339 19097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:29.869462 19097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:04:29.879438 19097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:39563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=38797
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:29.880848 19097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:29.882192 19097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:29.895730 19102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:29.895779 19105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:29.895939 19103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:29.896989 19097 server_base.cc:1034] running on GCE node
I20260620 15:04:29.897733 19097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:29.898738 19097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:29.899950 19097 hybrid_clock.cc:648] HybridClock initialized: now 1781967869899893 us; error 39 us; skew 500 ppm
I20260620 15:04:29.900276 19097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:29 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967869, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:29.903513 19097 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:04:29.904997 19097 webserver.cc:460] Webserver started at http://127.2.70.2:38797/ using document root <none> and password file <none>
I20260620 15:04:29.905846 19097 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:29.906005 19097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:29.911428 19097 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 15:04:29.914173 19112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:29.915472 19097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:29.915648 19097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "5105d1414cb245b79dfbdf04d09cb863"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "982282d0e55b287d7f58bfa2816b20cb"
server_key_iv: "811f3c78c14f83464710b70f1fd7b13d"
server_key_version: "encryptionkey@0"
I20260620 15:04:29.916635 19097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:29.935397 19097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:29.940177 19097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:29.940477 19097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:29.941624 19097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:29.942973 19097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:29.943100 19097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:29.943239 19097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:29.943333 19097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:29.977751 19097 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:39563
I20260620 15:04:29.977854 19224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:39563 every 8 connection(s)
I20260620 15:04:29.979226 19097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:04:29.980373 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 19097
I20260620 15:04:29.981132 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:34629
--local_ip_for_outbound_sockets=127.2.70.3
--tserver_master_addrs=127.2.70.62:34899
--webserver_port=39259
--webserver_interface=127.2.70.3
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:04:29.984658 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:29.982622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:34481)
0620 15:04:29.983474 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:29.983715 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":368,"threads_started":1}
W20260620 15:04:29.985438 19225 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:34899 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.62:34899: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
I20260620 15:04:29.986378 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:29.985990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:39921)
0620 15:04:29.986147 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:29.986225 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 15:04:29.987413 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:29.986968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:52405)
0620 15:04:29.987102 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:29.987213 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:04:29.987720 19225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 15:04:30.316012 19228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:30.316296 19228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:30.316435 19228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:30.323983 19228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:30.324103 19228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:30.324288 19228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:04:30.333855 19228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:34629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=39259
--tserver_master_addrs=127.2.70.62:34899
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:30.335170 19228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:30.336493 19228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:30.350167 19234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:30.350137 19236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:30.350211 19233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:30.354974 19228 server_base.cc:1034] running on GCE node
I20260620 15:04:30.355643 19228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:30.356637 19228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:30.357853 19228 hybrid_clock.cc:648] HybridClock initialized: now 1781967870357792 us; error 57 us; skew 500 ppm
I20260620 15:04:30.358178 19228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:30 dist-test-slave-n5vv krb5kdc[17777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967870, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:30.361366 19228 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:04:30.362797 19228 webserver.cc:460] Webserver started at http://127.2.70.3:39259/ using document root <none> and password file <none>
I20260620 15:04:30.363453 19228 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:30.363550 19228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:30.368685 19228 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260620 15:04:30.371341 19243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:30.372660 19228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:04:30.372797 19228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "9a29005a18a84af99ad1f61aed3c0dfa"
format_stamp: "Formatted at 2026-06-20 15:03:44 on dist-test-slave-n5vv"
server_key: "0e8b5de7671595e8650a31106e30497b"
server_key_iv: "de81e59b6b90abca84a6fc7616cf79c4"
server_key_version: "encryptionkey@0"
I20260620 15:04:30.373829 19228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:30.397202 19228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:30.401875 19228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:30.402164 19228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:30.403223 19228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:30.404680 19228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:30.404778 19228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:30.404942 19228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:30.405032 19228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:30.439160 19228 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:34629
I20260620 15:04:30.439244 19355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:34629 every 8 connection(s)
I20260620 15:04:30.440726 19228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:04:30.445469 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 19228
I20260620 15:04:30.445807 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:30.443702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:59101)
0620 15:04:30.444436 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:30.444639 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":286,"threads_started":1}
W20260620 15:04:30.446448 19356 heartbeater.cc:644] Failed to heartbeat to 127.2.70.62:34899 (0 consecutive failures): Network error: Failed to ping master at 127.2.70.62:34899: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
I20260620 15:04:30.447394 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:30.447028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:37303)
0620 15:04:30.447186 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:30.447261 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 15:04:30.448202 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:30.447922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:56611)
0620 15:04:30.448032 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:30.448100 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 15:04:30.448467 19356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 15:04:30.525233 19360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:30.524451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:37723)
0620 15:04:30.524900 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:30.525003 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":210,"threads_started":1}
I20260620 15:04:30.989096 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:30.988253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:57517)
0620 15:04:30.988774 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:30.988874 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":221,"threads_started":1}
I20260620 15:04:31.449836 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:31.449009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:60333)
0620 15:04:31.449465 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:31.449615 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":210,"threads_started":1}
I20260620 15:04:31.526880 19363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:31.526195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:59795)
0620 15:04:31.526644 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:31.526722 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":236,"threads_started":1}
I20260620 15:04:31.991139 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:31.990130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:41563)
0620 15:04:31.990751 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:31.990880 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":491,"threads_started":1}
I20260620 15:04:32.451515 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:32.450790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:38463)
0620 15:04:32.451247 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:32.451350 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":230,"threads_started":1}
I20260620 15:04:32.528395 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:32.527670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:49553)
0620 15:04:32.528128 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:32.528208 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":251,"threads_started":1}
I20260620 15:04:32.992805 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:32.992103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:47625)
0620 15:04:32.992546 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:32.992632 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260620 15:04:33.453140 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:33.452436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:34603)
0620 15:04:33.452886 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:33.452969 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":236,"threads_started":1}
I20260620 15:04:33.530172 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:33.529394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:55267)
0620 15:04:33.529920 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:33.530005 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":291,"threads_started":1}
I20260620 15:04:33.994583 19370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:33.993846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:36735)
0620 15:04:33.994300 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:33.994382 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":226,"threads_started":1}
I20260620 15:04:34.454741 19371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:34.454046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:47655)
0620 15:04:34.454488 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:34.454567 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":240,"threads_started":1}
I20260620 15:04:34.531905 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:34.531141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:41521)
0620 15:04:34.531653 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:34.531737 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":301,"threads_started":1}
I20260620 15:04:34.996299 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:34.995572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:53121)
0620 15:04:34.996051 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:34.996133 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260620 15:04:35.456677 19374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:35.455809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:57487)
0620 15:04:35.456401 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:35.456500 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":285,"threads_started":1}
I20260620 15:04:35.533617 19375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:35.532849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:49363)
0620 15:04:35.533298 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:35.533384 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":229,"threads_started":1}
I20260620 15:04:35.998076 19376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:35.997198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:56987)
0620 15:04:35.997803 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:35.997895 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":240,"threads_started":1}
I20260620 15:04:36.458525 19377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:36.457789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:44797)
0620 15:04:36.458244 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:36.458359 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260620 15:04:36.535271 19378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:36.534491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:37561)
0620 15:04:36.534971 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:36.535052 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":215,"threads_started":1}
I20260620 15:04:37.000258 19379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:36.999167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:56203)
0620 15:04:36.999799 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:36.999912 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":262,"threads_started":1}
I20260620 15:04:37.460237 19380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:37.459439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:38707)
0620 15:04:37.459924 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:37.460008 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":213,"threads_started":1}
I20260620 15:04:37.537062 19381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:37.536190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:45679)
0620 15:04:37.536741 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:37.536835 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":281,"threads_started":1}
I20260620 15:04:38.002089 19382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:38.001282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:46709)
0620 15:04:38.001842 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:38.001929 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":337,"threads_started":1}
I20260620 15:04:38.462096 19383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:38.461265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:50777)
0620 15:04:38.461833 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:38.461924 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":320,"threads_started":1}
I20260620 15:04:38.538784 19384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:38.538023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:39193)
0620 15:04:38.538505 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:38.538603 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":218,"threads_started":1}
I20260620 15:04:39.003819 19385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:39.003083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:41777)
0620 15:04:39.003553 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:39.003644 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":222,"threads_started":1}
I20260620 15:04:39.463886 19386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:39.463113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:59927)
0620 15:04:39.463609 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:39.463698 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":241,"threads_started":1}
I20260620 15:04:39.540431 19387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:39.539742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:33973)
0620 15:04:39.540155 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:39.540237 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":220,"threads_started":1}
I20260620 15:04:40.005839 19388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:40.004981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:50535)
0620 15:04:40.005572 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:40.005657 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":330,"threads_started":1}
I20260620 15:04:40.465569 19389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:40.464816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:54913)
0620 15:04:40.465253 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:40.465336 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":246,"threads_started":1}
I20260620 15:04:40.542450 19390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:40.541449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:36529)
0620 15:04:40.542059 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:40.542146 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260620 15:04:41.007855 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:41.006955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:53957)
0620 15:04:41.007494 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:41.007613 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":239,"threads_started":1}
I20260620 15:04:41.467330 19392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:41.466560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:47161)
0620 15:04:41.467038 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:41.467132 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260620 15:04:41.544101 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:41.543429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:50983)
0620 15:04:41.543860 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:41.543944 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":225,"threads_started":1}
I20260620 15:04:42.009768 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:42.008881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:37191)
0620 15:04:42.009430 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:42.009558 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":220,"threads_started":1}
I20260620 15:04:42.469156 19395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:42.468425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:53827)
0620 15:04:42.468897 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:42.468997 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":223,"threads_started":1}
I20260620 15:04:42.545957 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:42.545088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:52089)
0620 15:04:42.545664 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:42.545750 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":263,"threads_started":1}
I20260620 15:04:43.011766 19397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:43.010844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:45079)
0620 15:04:43.011451 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:43.011551 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":294,"threads_started":1}
I20260620 15:04:43.470999 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:43.470211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:58395)
0620 15:04:43.470714 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:43.470807 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":253,"threads_started":1}
I20260620 15:04:43.547516 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:43.546797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:48571)
0620 15:04:43.547249 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:43.547332 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
I20260620 15:04:44.013733 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:44.012879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:51933)
0620 15:04:44.013412 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:44.013548 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":303,"threads_started":1}
I20260620 15:04:44.472607 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:44.471893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:51971)
0620 15:04:44.472329 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:44.472412 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":222,"threads_started":1}
I20260620 15:04:44.549409 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:44.548677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:55281)
0620 15:04:44.549156 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:44.549229 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":261,"threads_started":1}
I20260620 15:04:45.015648 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:45.014931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:36957)
0620 15:04:45.015375 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:45.015470 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260620 15:04:45.458505 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 18925
I20260620 15:04:45.474968 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:45.474005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:46113)
0620 15:04:45.474666 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:45.474769 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":399,"threads_started":1}
I20260620 15:04:45.551525 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:45.550600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.1:42317)
0620 15:04:45.551208 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:45.551322 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":322,"threads_started":1}
I20260620 15:04:46.018165 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:46.016810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.2:38653)
0620 15:04:46.017912 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:46.018008 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":984,"thread_start_us":853,"threads_started":1}
I20260620 15:04:46.216598 19404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:45.460156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52282 (local address 127.2.70.1:40489)
0620 15:04:45.460797 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:45.460816 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:46.194512 (+733696us) server_negotiation.cc:407] Connection header received
0620 15:04:46.196261 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:46.196272 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:46.196790 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:46.197111 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:46.198882 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:46.200374 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:46.201888 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:46.202666 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:46.202819 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:46.209133 (+ 6314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:46.209142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:46.209161 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:46.209214 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:46.212680 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:46.213668 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:46.213675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:46.213683 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:46.213897 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:46.214328 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:46.214331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:46.214333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:46.215171 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:46.215522 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:46.215913 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 15:04:46.216256 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":433,"threads_started":1}
I20260620 15:04:46.218040 19089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:46.303174 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 18925
I20260620 15:04:46.320726 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 19097
I20260620 15:04:46.337949 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:46.321728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.2.70.2:39563)
0620 15:04:46.322285 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:46.322299 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:46.322340 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:04:46.323975 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:46.323984 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:46.324430 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:46.324718 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:46.326085 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:46.327254 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:46.328450 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:46.329093 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:46.329226 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:46.331754 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:46.331761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:46.331775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:46.331810 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:46.334712 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:46.335453 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:46.335459 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:46.335465 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:46.335627 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:46.335976 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:46.335979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:46.335981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:46.336478 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:46.336699 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:46.337095 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 15:04:46.337652 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":259,"threads_started":1}
I20260620 15:04:46.339301 19220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:46.422658 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 19097
I20260620 15:04:46.440208 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 19228
I20260620 15:04:46.461722 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:46.441287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.2.70.3:34629)
0620 15:04:46.445913 (+ 4626us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:46.445926 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:46.445963 (+ 37us) server_negotiation.cc:407] Connection header received
0620 15:04:46.447445 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:46.447455 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:46.447871 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:46.448155 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:46.449472 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:46.450712 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:46.452049 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:46.452721 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:46.452855 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:46.455420 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:46.455427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:46.455442 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:46.455481 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:46.458700 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:46.459279 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:46.459285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:46.459291 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:46.459484 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:46.459800 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:46.459803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:46.459806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:46.460295 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:46.460559 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:46.460902 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 15:04:46.461360 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4464,"thread_start_us":4363,"threads_started":1}
I20260620 15:04:46.463101 19351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:46.549616 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 19228
I20260620 15:04:46.549669 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:46.548519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34899 (local address 127.2.70.3:60737)
0620 15:04:46.549081 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:46.549304 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.62:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":226,"threads_started":1}
2026-06-20T15:04:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65249 ms)
[----------] 30 tests from SecurityITest (297268 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:04:46 dist-test-slave-n5vv krb5kdc[19416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:04:46 dist-test-slave-n5vv krb5kdc[19416](info): set up 2 sockets
Jun 20 15:04:46 dist-test-slave-n5vv krb5kdc[19416](info): commencing operation
krb5kdc: starting...
W20260620 15:04:47.665650 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:04:47 dist-test-slave-n5vv krb5kdc[19416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:04:47.914964 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:36773
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:36773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:48.243072 19429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:48.243439 19429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:48.243620 19429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:48.250923 19429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:04:48.251078 19429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:48.251187 19429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:48.251304 19429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:04:48.251395 19429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:04:48.261039 19429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:36773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:36773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:48.262387 19429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:48.263715 19429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:48.275867 19434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:48.275830 19437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:48.275911 19435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:48.276831 19429 server_base.cc:1034] running on GCE node
I20260620 15:04:48.277787 19429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:48.279218 19429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:48.280464 19429 hybrid_clock.cc:648] HybridClock initialized: now 1781967888280420 us; error 52 us; skew 500 ppm
I20260620 15:04:48.280825 19429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:48 dist-test-slave-n5vv krb5kdc[19416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:48.283982 19429 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:04:48.285619 19429 webserver.cc:460] Webserver started at http://127.2.70.62:45735/ using document root <none> and password file <none>
I20260620 15:04:48.286298 19429 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:48.286444 19429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:48.286746 19429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:48.289744 19429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "f055ae025f2c4430adc3fd9340521752"
format_stamp: "Formatted at 2026-06-20 15:04:48 on dist-test-slave-n5vv"
server_key: "de6142a0e49f0534ac5407847feaade2"
server_key_iv: "a324fa2ce49ac95bdfdf429337c295cc"
server_key_version: "encryptionkey@0"
I20260620 15:04:48.290513 19429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "f055ae025f2c4430adc3fd9340521752"
format_stamp: "Formatted at 2026-06-20 15:04:48 on dist-test-slave-n5vv"
server_key: "de6142a0e49f0534ac5407847feaade2"
server_key_iv: "a324fa2ce49ac95bdfdf429337c295cc"
server_key_version: "encryptionkey@0"
I20260620 15:04:48.295570 19429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:04:48.298943 19444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:48.300153 19429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:04:48.300328 19429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "f055ae025f2c4430adc3fd9340521752"
format_stamp: "Formatted at 2026-06-20 15:04:48 on dist-test-slave-n5vv"
server_key: "de6142a0e49f0534ac5407847feaade2"
server_key_iv: "a324fa2ce49ac95bdfdf429337c295cc"
server_key_version: "encryptionkey@0"
I20260620 15:04:48.300529 19429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:48.314509 19429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:48.319199 19429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:48.319526 19429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:48.338186 19429 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:36773
I20260620 15:04:48.338209 19495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:36773 every 8 connection(s)
I20260620 15:04:48.339620 19429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:04:48.343415 19496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:48.349107 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 19429
I20260620 15:04:48.349320 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:04:48.350005 2328 external_mini_cluster.cc:1428] Setting key f44b688aceb52f1e867e2dae55c087c8
I20260620 15:04:48.354751 19496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752: Bootstrap starting.
I20260620 15:04:48.358122 19496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:48.359230 19496 log.cc:824] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:48.362362 19496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752: No bootstrap required, opened a new log
Jun 20 15:04:48 dist-test-slave-n5vv krb5kdc[19416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:48.370760 19496 raft_consensus.cc:348] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f055ae025f2c4430adc3fd9340521752" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36773 } }
I20260620 15:04:48.371192 19496 raft_consensus.cc:374] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:48.371325 19496 raft_consensus.cc:729] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f055ae025f2c4430adc3fd9340521752, State: Initialized, Role: FOLLOWER
I20260620 15:04:48.372025 19496 consensus_queue.cc:260] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f055ae025f2c4430adc3fd9340521752" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36773 } }
I20260620 15:04:48.372318 19496 raft_consensus.cc:388] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:04:48.372462 19496 raft_consensus.cc:482] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:04:48.372650 19496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:48.373442 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:48.352123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.2.70.62:36773)
0620 15:04:48.353001 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:48.353013 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:48.353056 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:04:48.354863 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:48.354921 (+ 58us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:48.355348 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:48.355768 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:48.357395 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:48.358904 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:48.360354 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:48.361056 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:48.361288 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:48.365595 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:48.365603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:48.365621 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:48.365664 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:48.368829 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:48.369597 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:48.369604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:48.369611 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:48.369779 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:48.370173 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:48.370178 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:48.370182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:48.370717 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:48.371035 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:48.371492 (+ 457us) server_negotiation.cc:299] Negotiation successful
0620 15:04:48.371986 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":421,"threads_started":1}
I20260620 15:04:48.375317 19496 raft_consensus.cc:504] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f055ae025f2c4430adc3fd9340521752" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36773 } }
I20260620 15:04:48.375799 19496 leader_election.cc:302] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f055ae025f2c4430adc3fd9340521752; no voters:
I20260620 15:04:48.376519 19496 leader_election.cc:288] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:04:48.376605 19501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:04:48.377336 19501 raft_consensus.cc:686] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [term 1 LEADER]: Becoming Leader. State: Replica: f055ae025f2c4430adc3fd9340521752, State: Running, Role: LEADER
I20260620 15:04:48.377991 19501 consensus_queue.cc:237] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f055ae025f2c4430adc3fd9340521752" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36773 } }
I20260620 15:04:48.378782 19496 sys_catalog.cc:564] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:04:48.381068 19503 sys_catalog.cc:455] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f055ae025f2c4430adc3fd9340521752. Latest consensus state: current_term: 1 leader_uuid: "f055ae025f2c4430adc3fd9340521752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f055ae025f2c4430adc3fd9340521752" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36773 } } }
I20260620 15:04:48.381119 19502 sys_catalog.cc:455] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f055ae025f2c4430adc3fd9340521752" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f055ae025f2c4430adc3fd9340521752" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36773 } } }
I20260620 15:04:48.381460 19503 sys_catalog.cc:458] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:48.381528 19502 sys_catalog.cc:458] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752 [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:48.383662 19508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:04:48.388345 19508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:04:48.397708 19508 catalog_manager.cc:1347] Generated new cluster ID: bcc9703d0ee7414683d49185fdadd2c2
I20260620 15:04:48.397846 19508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:04:48.436497 19508 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:04:48.437948 19508 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:04:48.455014 19508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752: Generated new TSK 0
I20260620 15:04:48.455974 19508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:04:48.526603 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36773
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:48.862468 19522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:48.862789 19522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:48.862962 19522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:48.870261 19522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:48.870424 19522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:48.870671 19522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:04:48.881407 19522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:48.882928 19522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:48.884335 19522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:48.898320 19527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:48.898283 19530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:48.898383 19528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:48.899060 19522 server_base.cc:1034] running on GCE node
I20260620 15:04:48.899662 19522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:48.900597 19522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:48.901849 19522 hybrid_clock.cc:648] HybridClock initialized: now 1781967888901794 us; error 60 us; skew 500 ppm
I20260620 15:04:48.902186 19522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:48 dist-test-slave-n5vv krb5kdc[19416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:48.905393 19522 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:04:48.906848 19522 webserver.cc:460] Webserver started at http://127.2.70.1:38521/ using document root <none> and password file <none>
I20260620 15:04:48.907545 19522 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:48.907685 19522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:48.908004 19522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:48.910904 19522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "56d61b3a16104872a0a5fafe931a38ce"
format_stamp: "Formatted at 2026-06-20 15:04:48 on dist-test-slave-n5vv"
server_key: "109d2f1c389bbd06af02d0b44a84be03"
server_key_iv: "bb7abb3b3f565f9c838a146b73a82ed7"
server_key_version: "encryptionkey@0"
I20260620 15:04:48.911682 19522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "56d61b3a16104872a0a5fafe931a38ce"
format_stamp: "Formatted at 2026-06-20 15:04:48 on dist-test-slave-n5vv"
server_key: "109d2f1c389bbd06af02d0b44a84be03"
server_key_iv: "bb7abb3b3f565f9c838a146b73a82ed7"
server_key_version: "encryptionkey@0"
I20260620 15:04:48.916854 19522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 15:04:48.920449 19537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:48.921860 19522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:04:48.922043 19522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "56d61b3a16104872a0a5fafe931a38ce"
format_stamp: "Formatted at 2026-06-20 15:04:48 on dist-test-slave-n5vv"
server_key: "109d2f1c389bbd06af02d0b44a84be03"
server_key_iv: "bb7abb3b3f565f9c838a146b73a82ed7"
server_key_version: "encryptionkey@0"
I20260620 15:04:48.922305 19522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:48.937408 19522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:48.941280 19522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:48.941628 19522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:48.942613 19522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:48.943930 19522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:48.944033 19522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:48.944180 19522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:48.944257 19522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:48.977901 19522 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:35621
I20260620 15:04:48.977945 19649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:35621 every 8 connection(s)
I20260620 15:04:48.979338 19522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:04:48.983506 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 19522
I20260620 15:04:48.983755 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:04:48.984448 2328 external_mini_cluster.cc:1428] Setting key 3ab7053612b1972c8528fa9e60ae9429
Jun 20 15:04:48 dist-test-slave-n5vv krb5kdc[19416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967888, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:49.002612 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:48.982043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:58495 (local address 127.2.70.62:36773)
0620 15:04:48.982679 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:48.982683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:48.983428 (+ 745us) server_negotiation.cc:407] Connection header received
0620 15:04:48.985519 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:48.985527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:48.985604 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:48.985722 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:48.987467 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:48.988204 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:48.990344 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:48.991090 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:48.991243 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:48.995339 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:48.995348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:48.995356 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:48.995404 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:48.999024 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:48.999869 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:48.999878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:48.999884 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:49.000048 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:49.000481 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:49.000488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:49.000494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:49.000890 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:49.001047 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:49.001843 (+ 796us) server_negotiation.cc:299] Negotiation successful
0620 15:04:49.002114 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":378,"threads_started":1}
I20260620 15:04:49.003373 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:48.982441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:36773 (local address 127.2.70.1:58495)
0620 15:04:48.983244 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:48.983298 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:48.985235 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:48.985990 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:48.986001 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:48.986727 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:48.987293 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:48.987313 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:48.988367 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:48.988379 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:48.990166 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:48.990184 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:48.991322 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:48.991330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:48.991681 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:48.992502 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:48.992536 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:48.995113 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:48.999226 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:48.999234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:48.999250 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:48.999729 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:49.000165 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:49.000168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:49.000171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:49.000362 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:49.001012 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:49.001018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:49.001378 (+ 360us) client_negotiation.cc:769] Sending connection context
0620 15:04:49.001683 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 15:04:49.002122 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":355,"threads_started":1}
I20260620 15:04:49.005190 19650 heartbeater.cc:347] Connected to a master server at 127.2.70.62:36773
I20260620 15:04:49.005568 19650 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:49.006520 19650 heartbeater.cc:511] Master 127.2.70.62:36773 requested a full tablet report, sending...
I20260620 15:04:49.009249 19461 ts_manager.cc:194] Registered new tserver with Master: 56d61b3a16104872a0a5fafe931a38ce (127.2.70.1:35621)
I20260620 15:04:49.011531 19461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:58495
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:04:49.042909 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36773
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:49.368888 19656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:49.369158 19656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:49.369346 19656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:49.376650 19656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:49.376758 19656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:49.376955 19656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:04:49.387079 19656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:49.388384 19656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:49.389842 19656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:49.403095 19661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:49.403118 19662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:49.403050 19664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:49.404189 19656 server_base.cc:1034] running on GCE node
I20260620 15:04:49.404857 19656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:49.405900 19656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:49.407155 19656 hybrid_clock.cc:648] HybridClock initialized: now 1781967889407107 us; error 32 us; skew 500 ppm
I20260620 15:04:49.407533 19656 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:49 dist-test-slave-n5vv krb5kdc[19416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:49.410885 19656 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:04:49.412402 19656 webserver.cc:460] Webserver started at http://127.2.70.2:38545/ using document root <none> and password file <none>
I20260620 15:04:49.413132 19656 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:49.413234 19656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:49.413635 19656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:49.416625 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "2657120de7524546b0a13a170fabf953"
format_stamp: "Formatted at 2026-06-20 15:04:49 on dist-test-slave-n5vv"
server_key: "c198461274857649b032b493027f6df9"
server_key_iv: "ac8c819179578b8e7b5264a195da6bd5"
server_key_version: "encryptionkey@0"
I20260620 15:04:49.417464 19656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "2657120de7524546b0a13a170fabf953"
format_stamp: "Formatted at 2026-06-20 15:04:49 on dist-test-slave-n5vv"
server_key: "c198461274857649b032b493027f6df9"
server_key_iv: "ac8c819179578b8e7b5264a195da6bd5"
server_key_version: "encryptionkey@0"
I20260620 15:04:49.422855 19656 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:04:49.426411 19671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:49.427795 19656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260620 15:04:49.427969 19656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "2657120de7524546b0a13a170fabf953"
format_stamp: "Formatted at 2026-06-20 15:04:49 on dist-test-slave-n5vv"
server_key: "c198461274857649b032b493027f6df9"
server_key_iv: "ac8c819179578b8e7b5264a195da6bd5"
server_key_version: "encryptionkey@0"
I20260620 15:04:49.428170 19656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:49.442857 19656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:49.447082 19656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:49.447340 19656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:49.448275 19656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:49.449617 19656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:49.449690 19656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:49.449782 19656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:49.449831 19656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:49.483290 19656 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:39299
I20260620 15:04:49.483326 19783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:39299 every 8 connection(s)
I20260620 15:04:49.484754 19656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:04:49.486488 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 19656
I20260620 15:04:49.486668 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:04:49.487142 2328 external_mini_cluster.cc:1428] Setting key ebb26c385eaf5c639a189eb9285547d3
Jun 20 15:04:49 dist-test-slave-n5vv krb5kdc[19416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:49.504995 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:49.487476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:37301 (local address 127.2.70.62:36773)
0620 15:04:49.487680 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:49.487686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:49.489022 (+ 1336us) server_negotiation.cc:407] Connection header received
0620 15:04:49.490995 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:49.491001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:49.491094 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:49.491202 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:49.492721 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:49.493412 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:49.494981 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:49.495393 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:49.495537 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:49.499186 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:49.499191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:49.499194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:49.499226 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:49.501825 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:49.502574 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:49.502580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:49.502584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:49.502712 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:49.503136 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:49.503142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:49.503146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:49.503405 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:49.503617 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:49.504410 (+ 793us) server_negotiation.cc:299] Negotiation successful
0620 15:04:49.504649 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 15:04:49.506032 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:49.488051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:36773 (local address 127.2.70.2:37301)
0620 15:04:49.488851 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:49.488910 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:49.490722 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:49.491443 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:49.491452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:49.491959 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:49.492529 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:49.492567 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:49.493582 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:49.493592 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:49.494828 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:49.494840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:49.495615 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:49.495622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:49.495856 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:49.496558 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:49.496580 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:49.498961 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:49.501966 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:49.501975 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:49.501991 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:49.502436 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:49.502820 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:49.502826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:49.502830 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:49.503025 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:49.503541 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:49.503573 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:49.503955 (+ 382us) client_negotiation.cc:769] Sending connection context
0620 15:04:49.504286 (+ 331us) client_negotiation.cc:240] Negotiation successful
0620 15:04:49.504821 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":353,"threads_started":1}
I20260620 15:04:49.507644 19784 heartbeater.cc:347] Connected to a master server at 127.2.70.62:36773
I20260620 15:04:49.508005 19784 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:49.508736 19784 heartbeater.cc:511] Master 127.2.70.62:36773 requested a full tablet report, sending...
I20260620 15:04:49.510336 19461 ts_manager.cc:194] Registered new tserver with Master: 2657120de7524546b0a13a170fabf953 (127.2.70.2:39299)
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:04:49.511584 19461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:37301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:04:49.543092 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36773
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:49.875865 19789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:49.876148 19789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:49.876318 19789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:49.884066 19789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:49.884186 19789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:49.884383 19789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:04:49.894330 19789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:36773
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:49.895684 19789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:49.897078 19789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:49.910853 19794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:49.910840 19795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:49.910866 19797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:49.911142 19789 server_base.cc:1034] running on GCE node
I20260620 15:04:49.912078 19789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:49.913043 19789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:49.914283 19789 hybrid_clock.cc:648] HybridClock initialized: now 1781967889914231 us; error 39 us; skew 500 ppm
I20260620 15:04:49.914659 19789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:49 dist-test-slave-n5vv krb5kdc[19416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:49.918057 19789 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:04:49.919569 19789 webserver.cc:460] Webserver started at http://127.2.70.3:39279/ using document root <none> and password file <none>
I20260620 15:04:49.920267 19789 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:49.920418 19789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:49.920735 19789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:49.923712 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "31db78a70bc84ac0be1ca5832a49a163"
format_stamp: "Formatted at 2026-06-20 15:04:49 on dist-test-slave-n5vv"
server_key: "fdfca2e41dadf059fbc12d17493b05d4"
server_key_iv: "fe7193e66a32f939fcf6116d75376e3b"
server_key_version: "encryptionkey@0"
I20260620 15:04:49.924484 19789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "31db78a70bc84ac0be1ca5832a49a163"
format_stamp: "Formatted at 2026-06-20 15:04:49 on dist-test-slave-n5vv"
server_key: "fdfca2e41dadf059fbc12d17493b05d4"
server_key_iv: "fe7193e66a32f939fcf6116d75376e3b"
server_key_version: "encryptionkey@0"
I20260620 15:04:49.930766 19789 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260620 15:04:49.934532 19804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:49.935853 19789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:04:49.936030 19789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "31db78a70bc84ac0be1ca5832a49a163"
format_stamp: "Formatted at 2026-06-20 15:04:49 on dist-test-slave-n5vv"
server_key: "fdfca2e41dadf059fbc12d17493b05d4"
server_key_iv: "fe7193e66a32f939fcf6116d75376e3b"
server_key_version: "encryptionkey@0"
I20260620 15:04:49.936228 19789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:49.957667 19789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:49.961915 19789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:49.962209 19789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:49.963189 19789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:49.964500 19789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:49.964607 19789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:49.964761 19789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:49.964835 19789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:49.999334 19789 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:46171
I20260620 15:04:49.999387 19916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:46171 every 8 connection(s)
I20260620 15:04:50.000795 19789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:04:50.008037 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 19789
I20260620 15:04:50.008219 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:04:50.008756 2328 external_mini_cluster.cc:1428] Setting key d7d688ce3787da73d1eb073d63112ffe
Jun 20 15:04:50 dist-test-slave-n5vv krb5kdc[19416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967889, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:50.015146 19650 heartbeater.cc:503] Master 127.2.70.62:36773 was elected leader, sending a full tablet report...
I20260620 15:04:50.021426 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.003729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:35737 (local address 127.2.70.62:36773)
0620 15:04:50.003959 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.003967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.005114 (+ 1147us) server_negotiation.cc:407] Connection header received
0620 15:04:50.007116 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.007122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.007225 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.007347 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:50.009068 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.009992 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.011536 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.011922 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.012055 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.015580 (+ 3525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:50.015585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:50.015589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:50.015622 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:50.018349 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.019104 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.019110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.019114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.019245 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.019694 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.019700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.019704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.019970 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:50.020125 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.020830 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.021052 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 15:04:50.022433 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.004165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:36773 (local address 127.2.70.3:35737)
0620 15:04:50.004919 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:50.004958 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:50.006833 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:50.007587 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:50.007595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:50.008154 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:50.008890 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.008917 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.010147 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.010154 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.011393 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.011404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.012142 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.012153 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.012403 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.013102 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:50.013125 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:50.015370 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:50.018522 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:50.018532 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:50.018549 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:50.018981 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:50.019364 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:50.019369 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:50.019373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:50.019577 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:50.020100 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:50.020108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:50.020448 (+ 340us) client_negotiation.cc:769] Sending connection context
0620 15:04:50.020729 (+ 281us) client_negotiation.cc:240] Negotiation successful
0620 15:04:50.021284 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":269,"threads_started":1}
I20260620 15:04:50.023998 19917 heartbeater.cc:347] Connected to a master server at 127.2.70.62:36773
I20260620 15:04:50.024326 19917 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:50.025019 19917 heartbeater.cc:511] Master 127.2.70.62:36773 requested a full tablet report, sending...
I20260620 15:04:50.026463 19461 ts_manager.cc:194] Registered new tserver with Master: 31db78a70bc84ac0be1ca5832a49a163 (127.2.70.3:46171)
I20260620 15:04:50.027360 19461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:35737
I20260620 15:04:50.036140 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:04:50.053632 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.040996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.2.70.62:36773)
0620 15:04:50.041164 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.041169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.041677 (+ 508us) server_negotiation.cc:407] Connection header received
0620 15:04:50.041869 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.041873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.041940 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.042018 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:50.043488 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.044201 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.045563 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.045956 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.046050 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.048447 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:50.048452 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:50.048455 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:50.048486 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:50.051089 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.051736 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.051740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.051743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.051873 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.052220 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.052223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.052225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.052582 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:50.052777 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.053106 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.053322 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:04:50.057674 19461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:04:50.064471 19461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:04:50.088207 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.076463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44982 (local address 127.2.70.3:46171)
0620 15:04:50.078703 (+ 2240us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.078715 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.078761 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:04:50.078878 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.078887 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.079193 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.079467 (+ 274us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:50.080776 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.082474 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.085331 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.086926 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.087083 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.087347 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.087677 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.087858 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2033,"thread_start_us":1888,"threads_started":1}
I20260620 15:04:50.088245 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.076347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:46171 (local address 127.0.0.1:44982)
0620 15:04:50.077578 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:50.077637 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:50.077852 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:50.079898 (+ 2046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:50.079909 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:50.079954 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:50.080527 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.080545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.082642 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.082652 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.085128 (+ 2476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.085144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.087156 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.087167 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.087442 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.087483 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:04:50.087621 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:04:50.087826 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":1,"thread_start_us":723,"threads_started":1}
I20260620 15:04:50.089730 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.078003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35621 (local address 127.0.0.1:55976)
0620 15:04:50.080721 (+ 2718us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:50.080746 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:50.080876 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:50.081805 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:50.081812 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:50.081839 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:50.082350 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.082362 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.083984 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.083991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.087233 (+ 3242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.087249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.088848 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.088855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.089018 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.089052 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:04:50.089176 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 15:04:50.089307 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2561,"mutex_wait_us":2,"thread_start_us":2379,"threads_started":1}
I20260620 15:04:50.091540 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.077437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.2.70.2:39299)
0620 15:04:50.081313 (+ 3876us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.081323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.081352 (+ 29us) server_negotiation.cc:407] Connection header received
0620 15:04:50.081467 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.081474 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.081716 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.081900 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:50.082849 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.084640 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.089023 (+ 4383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.090559 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.090692 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.090887 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.091171 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.091307 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3680,"thread_start_us":3537,"threads_started":1}
I20260620 15:04:50.091549 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.076832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:39299 (local address 127.0.0.1:49596)
0620 15:04:50.079042 (+ 2210us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:50.079075 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:50.079198 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:50.082131 (+ 2933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:50.082137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:50.082160 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:50.082613 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.082627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.084793 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.084800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.088834 (+ 4034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.088847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.090790 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.090797 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.090957 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.090988 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:04:50.091076 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 15:04:50.091193 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2063,"mutex_wait_us":1463,"thread_start_us":1938,"threads_started":1}
I20260620 15:04:50.092567 19852 tablet_service.cc:1467] Processing CreateTablet for tablet 135785df7a5d46f1a95b3f355c13df01 (DEFAULT_TABLE table=test-table [id=a98f055f0aa04f94b24ba0d85bc9f7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:50.093825 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.078557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55976 (local address 127.2.70.1:35621)
0620 15:04:50.079968 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.079981 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.080873 (+ 892us) server_negotiation.cc:407] Connection header received
0620 15:04:50.081045 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.081053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.081324 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.081560 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:50.082572 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.083777 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.087398 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.088653 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.088786 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.089036 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.089223 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.089384 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1165,"thread_start_us":1012,"threads_started":1}
I20260620 15:04:50.093921 19852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135785df7a5d46f1a95b3f355c13df01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:50.096483 19719 tablet_service.cc:1467] Processing CreateTablet for tablet 135785df7a5d46f1a95b3f355c13df01 (DEFAULT_TABLE table=test-table [id=a98f055f0aa04f94b24ba0d85bc9f7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:50.097698 19719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135785df7a5d46f1a95b3f355c13df01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:50.097560 19585 tablet_service.cc:1467] Processing CreateTablet for tablet 135785df7a5d46f1a95b3f355c13df01 (DEFAULT_TABLE table=test-table [id=a98f055f0aa04f94b24ba0d85bc9f7f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:50.098872 19585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 135785df7a5d46f1a95b3f355c13df01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:50.108630 19935 tablet_bootstrap.cc:492] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Bootstrap starting.
I20260620 15:04:50.110070 19936 tablet_bootstrap.cc:492] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Bootstrap starting.
I20260620 15:04:50.113174 19937 tablet_bootstrap.cc:492] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Bootstrap starting.
I20260620 15:04:50.113631 19936 tablet_bootstrap.cc:654] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:50.113790 19935 tablet_bootstrap.cc:654] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:50.114970 19936 log.cc:824] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:50.114967 19935 log.cc:824] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:50.116341 19937 tablet_bootstrap.cc:654] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:50.117386 19937 log.cc:824] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:50.118345 19935 tablet_bootstrap.cc:492] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: No bootstrap required, opened a new log
I20260620 15:04:50.118371 19936 tablet_bootstrap.cc:492] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: No bootstrap required, opened a new log
I20260620 15:04:50.118655 19936 ts_tablet_manager.cc:1397] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260620 15:04:50.118655 19935 ts_tablet_manager.cc:1397] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 15:04:50.120548 19937 tablet_bootstrap.cc:492] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: No bootstrap required, opened a new log
I20260620 15:04:50.120921 19937 ts_tablet_manager.cc:1397] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260620 15:04:50.128453 19937 raft_consensus.cc:348] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.128863 19937 raft_consensus.cc:374] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:50.128969 19937 raft_consensus.cc:729] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d61b3a16104872a0a5fafe931a38ce, State: Initialized, Role: FOLLOWER
I20260620 15:04:50.129695 19937 consensus_queue.cc:260] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.129751 19936 raft_consensus.cc:348] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.129751 19935 raft_consensus.cc:348] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.130182 19935 raft_consensus.cc:374] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:50.130182 19936 raft_consensus.cc:374] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:50.130285 19935 raft_consensus.cc:729] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2657120de7524546b0a13a170fabf953, State: Initialized, Role: FOLLOWER
I20260620 15:04:50.130285 19936 raft_consensus.cc:729] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31db78a70bc84ac0be1ca5832a49a163, State: Initialized, Role: FOLLOWER
I20260620 15:04:50.130880 19935 consensus_queue.cc:260] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.130908 19936 consensus_queue.cc:260] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.135066 19937 ts_tablet_manager.cc:1428] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260620 15:04:50.135140 19917 heartbeater.cc:503] Master 127.2.70.62:36773 was elected leader, sending a full tablet report...
I20260620 15:04:50.135195 19784 heartbeater.cc:503] Master 127.2.70.62:36773 was elected leader, sending a full tablet report...
I20260620 15:04:50.136656 19935 ts_tablet_manager.cc:1428] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Time spent starting tablet: real 0.018s user 0.004s sys 0.013s
I20260620 15:04:50.136935 19936 ts_tablet_manager.cc:1428] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Time spent starting tablet: real 0.018s user 0.007s sys 0.010s
I20260620 15:04:50.176960 19941 raft_consensus.cc:482] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:04:50.177320 19941 raft_consensus.cc:504] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.179123 19941 leader_election.cc:288] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31db78a70bc84ac0be1ca5832a49a163 (127.2.70.3:46171), 2657120de7524546b0a13a170fabf953 (127.2.70.2:39299)
I20260620 15:04:50.188433 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.179658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:51317 (local address 127.2.70.2:39299)
0620 15:04:50.179890 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.179898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.180755 (+ 857us) server_negotiation.cc:407] Connection header received
0620 15:04:50.181040 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.181048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.181135 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.181234 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:50.182096 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.182821 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.185975 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.187231 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.187366 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.187469 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.188027 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.188186 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:04:50.188458 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.179671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:37787 (local address 127.2.70.3:46171)
0620 15:04:50.179888 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.179896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.181072 (+ 1176us) server_negotiation.cc:407] Connection header received
0620 15:04:50.181344 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.181349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.181428 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.181521 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:50.182250 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.182972 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.185975 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.187230 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.187366 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.187488 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.188027 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.188169 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260620 15:04:50.188659 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.179656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:39299 (local address 127.2.70.1:51317)
0620 15:04:50.180628 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:50.180656 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:50.180888 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:50.181353 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:50.181358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:50.181387 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:50.181923 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.181938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.182954 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.182962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.185793 (+ 2831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.185809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.187520 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.187528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.187800 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.187875 (+ 75us) client_negotiation.cc:769] Sending connection context
0620 15:04:50.187976 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:04:50.188154 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":238,"thread_start_us":493,"threads_started":1}
I20260620 15:04:50.189103 19945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.179688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:46171 (local address 127.2.70.1:37787)
0620 15:04:50.180976 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:50.180994 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:50.181125 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:50.181730 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:50.181733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:50.181751 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:50.182098 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.182106 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.183138 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.183144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.185800 (+ 2656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:50.185809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.187539 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.187545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:50.187804 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.187884 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 15:04:50.187976 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:04:50.188157 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"mutex_wait_us":296,"thread_start_us":878,"threads_started":1}
I20260620 15:04:50.189759 19739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135785df7a5d46f1a95b3f355c13df01" candidate_uuid: "56d61b3a16104872a0a5fafe931a38ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2657120de7524546b0a13a170fabf953" is_pre_election: true
I20260620 15:04:50.190109 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135785df7a5d46f1a95b3f355c13df01" candidate_uuid: "56d61b3a16104872a0a5fafe931a38ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31db78a70bc84ac0be1ca5832a49a163" is_pre_election: true
I20260620 15:04:50.190233 19739 raft_consensus.cc:2454] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56d61b3a16104872a0a5fafe931a38ce in term 0.
I20260620 15:04:50.190605 19872 raft_consensus.cc:2454] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56d61b3a16104872a0a5fafe931a38ce in term 0.
I20260620 15:04:50.190889 19540 leader_election.cc:302] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2657120de7524546b0a13a170fabf953, 56d61b3a16104872a0a5fafe931a38ce; no voters:
I20260620 15:04:50.191340 19941 raft_consensus.cc:2781] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:04:50.191488 19941 raft_consensus.cc:482] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:04:50.191608 19941 raft_consensus.cc:3037] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:50.193965 19941 raft_consensus.cc:504] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.194677 19941 leader_election.cc:288] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [CANDIDATE]: Term 1 election: Requested vote from peers 31db78a70bc84ac0be1ca5832a49a163 (127.2.70.3:46171), 2657120de7524546b0a13a170fabf953 (127.2.70.2:39299)
I20260620 15:04:50.195055 19872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135785df7a5d46f1a95b3f355c13df01" candidate_uuid: "56d61b3a16104872a0a5fafe931a38ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31db78a70bc84ac0be1ca5832a49a163"
I20260620 15:04:50.195189 19739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "135785df7a5d46f1a95b3f355c13df01" candidate_uuid: "56d61b3a16104872a0a5fafe931a38ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2657120de7524546b0a13a170fabf953"
I20260620 15:04:50.195271 19872 raft_consensus.cc:3037] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:50.195444 19739 raft_consensus.cc:3037] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:50.197636 19872 raft_consensus.cc:2454] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56d61b3a16104872a0a5fafe931a38ce in term 1.
I20260620 15:04:50.197717 19739 raft_consensus.cc:2454] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56d61b3a16104872a0a5fafe931a38ce in term 1.
I20260620 15:04:50.198140 19539 leader_election.cc:302] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31db78a70bc84ac0be1ca5832a49a163, 56d61b3a16104872a0a5fafe931a38ce; no voters:
I20260620 15:04:50.198495 19941 raft_consensus.cc:2781] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:04:50.198992 19941 raft_consensus.cc:686] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 1 LEADER]: Becoming Leader. State: Replica: 56d61b3a16104872a0a5fafe931a38ce, State: Running, Role: LEADER
I20260620 15:04:50.199599 19941 consensus_queue.cc:237] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } }
I20260620 15:04:50.204321 19460 catalog_manager.cc:5495] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 56d61b3a16104872a0a5fafe931a38ce (127.2.70.1). New cstate: current_term: 1 leader_uuid: "56d61b3a16104872a0a5fafe931a38ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56d61b3a16104872a0a5fafe931a38ce" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 } health_report { overall_health: UNKNOWN } } }
W20260620 15:04:50.231696 19651 tablet.cc:2369] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 15:04:50.236814 19785 tablet.cc:2369] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:04:50.249197 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.243664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55986 (local address 127.2.70.1:35621)
0620 15:04:50.243881 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.243886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.243904 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:04:50.244077 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.244080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.244158 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.244282 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:04:50.244659 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.245337 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.247040 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.247487 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.247589 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.248091 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:04:50.248610 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:04:50.248725 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.248828 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.248894 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260620 15:04:50.252978 19918 tablet.cc:2369] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:04:50.257251 19739 raft_consensus.cc:1261] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 1 FOLLOWER]: Refusing update from remote peer 56d61b3a16104872a0a5fafe931a38ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:04:50.257244 19872 raft_consensus.cc:1261] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 1 FOLLOWER]: Refusing update from remote peer 56d61b3a16104872a0a5fafe931a38ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:04:50.258234 19946 consensus_queue.cc:1035] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "31db78a70bc84ac0be1ca5832a49a163" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:04:50.258592 19941 consensus_queue.cc:1035] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2657120de7524546b0a13a170fabf953" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:04:50.275285 19950 mvcc.cc:205] Tried to move back new op lower bound from 7298940478484434944 to 7298940478261436416. Current Snapshot: MvccSnapshot[applied={T|T < 7298940478484434944}]
I20260620 15:04:50.285634 19460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:04:50.285948 19460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:04:50.289268 19460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f055ae025f2c4430adc3fd9340521752: Sending DeleteTablet for 3 replicas of tablet 135785df7a5d46f1a95b3f355c13df01
I20260620 15:04:50.290593 19852 tablet_service.cc:1514] Processing DeleteTablet for tablet 135785df7a5d46f1a95b3f355c13df01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:50 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:44982
I20260620 15:04:50.290594 19585 tablet_service.cc:1514] Processing DeleteTablet for tablet 135785df7a5d46f1a95b3f355c13df01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:50 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:55976
I20260620 15:04:50.291386 19719 tablet_service.cc:1514] Processing DeleteTablet for tablet 135785df7a5d46f1a95b3f355c13df01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:50 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:49596
I20260620 15:04:50.291544 19960 tablet_replica.cc:330] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: stopping tablet replica
I20260620 15:04:50.292080 19960 raft_consensus.cc:2229] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:04:50.292862 19960 raft_consensus.cc:2258] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:50.293881 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 19522
I20260620 15:04:50.295686 19960 ts_tablet_manager.cc:1905] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:50.302006 19959 tablet_replica.cc:330] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: stopping tablet replica
I20260620 15:04:50.302552 19959 raft_consensus.cc:2229] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:04:50.302954 19961 tablet_replica.cc:330] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: stopping tablet replica
I20260620 15:04:50.303223 19959 raft_consensus.cc:2258] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:50.303406 19961 raft_consensus.cc:2229] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:04:50.303930 19961 raft_consensus.cc:2258] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:50.305857 19959 ts_tablet_manager.cc:1905] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:50.306228 19961 ts_tablet_manager.cc:1905] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:50.311555 19960 ts_tablet_manager.cc:1918] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:50.311749 19960 log.cc:1196] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/135785df7a5d46f1a95b3f355c13df01
I20260620 15:04:50.312393 19960 ts_tablet_manager.cc:1939] T 135785df7a5d46f1a95b3f355c13df01 P 56d61b3a16104872a0a5fafe931a38ce: Deleting consensus metadata
I20260620 15:04:50.312983 19959 ts_tablet_manager.cc:1918] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:50.313203 19959 log.cc:1196] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/135785df7a5d46f1a95b3f355c13df01
I20260620 15:04:50.313799 19961 ts_tablet_manager.cc:1918] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:50.313831 19959 ts_tablet_manager.cc:1939] T 135785df7a5d46f1a95b3f355c13df01 P 31db78a70bc84ac0be1ca5832a49a163: Deleting consensus metadata
I20260620 15:04:50.314026 19961 log.cc:1196] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/135785df7a5d46f1a95b3f355c13df01
I20260620 15:04:50.314520 19446 catalog_manager.cc:4841] TS 56d61b3a16104872a0a5fafe931a38ce (127.2.70.1:35621): tablet 135785df7a5d46f1a95b3f355c13df01 (table test-table [id=a98f055f0aa04f94b24ba0d85bc9f7f6]) successfully deleted
I20260620 15:04:50.314625 19961 ts_tablet_manager.cc:1939] T 135785df7a5d46f1a95b3f355c13df01 P 2657120de7524546b0a13a170fabf953: Deleting consensus metadata
I20260620 15:04:50.315732 19446 catalog_manager.cc:4841] TS 31db78a70bc84ac0be1ca5832a49a163 (127.2.70.3:46171): tablet 135785df7a5d46f1a95b3f355c13df01 (table test-table [id=a98f055f0aa04f94b24ba0d85bc9f7f6]) successfully deleted
I20260620 15:04:50.316484 19447 catalog_manager.cc:4841] TS 2657120de7524546b0a13a170fabf953 (127.2.70.2:39299): tablet 135785df7a5d46f1a95b3f355c13df01 (table test-table [id=a98f055f0aa04f94b24ba0d85bc9f7f6]) successfully deleted
Jun 20 15:04:50 dist-test-slave-n5vv krb5kdc[19416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:04:50.326206 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.296284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55988 (local address 127.2.70.1:35621)
0620 15:04:50.296506 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.296516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.302241 (+ 5725us) server_negotiation.cc:407] Connection header received
0620 15:04:50.302474 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.302483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.302586 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.302714 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:50.304719 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.305959 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.307944 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.308630 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.308747 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.318936 (+ 10189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:50.318956 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:50.318981 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:50.319042 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:50.323151 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.323904 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.323922 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.323925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.324107 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.324483 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.324488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.324493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.324970 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:50.325263 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.325592 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.325830 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:04:50.326972 19645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:50.438696 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 19522
I20260620 15:04:50.457707 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 19656
Jun 20 15:04:50 dist-test-slave-n5vv krb5kdc[19416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:04:50.472411 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.458621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49612 (local address 127.2.70.2:39299)
0620 15:04:50.458787 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.458791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.458808 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:04:50.458911 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.458916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.458982 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.459067 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:50.460328 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.461039 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.462685 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.463067 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.463160 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.467295 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:50.467308 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:50.467324 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:50.467360 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:50.470053 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.470687 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.470694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.470696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.470812 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.471113 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.471116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.471118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.471477 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:50.471690 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.471963 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.472143 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:04:50.473151 19779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:50.568168 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 19656
I20260620 15:04:50.585558 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 19789
Jun 20 15:04:50 dist-test-slave-n5vv krb5kdc[19416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:04:50.600659 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:50.586286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.2.70.3:46171)
0620 15:04:50.586456 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:50.586461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:50.586527 (+ 66us) server_negotiation.cc:407] Connection header received
0620 15:04:50.586674 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:50.586678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:50.586745 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:50.586845 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:50.588086 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.588820 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.590187 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:50.590644 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:50.590739 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:50.594932 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:50.594948 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:50.594965 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:50.594999 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:50.597705 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.598683 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.598691 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.598693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.598853 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:50.599331 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:50.599334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:50.599336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:50.599690 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:50.599912 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:50.600203 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 15:04:50.600379 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 15:04:50.601315 19912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:50.691165 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 19789
I20260620 15:04:50.707737 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 19429
I20260620 15:04:50.708873 19491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:50.821039 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 19429
2026-06-20T15:04:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4275 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T15:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:04:50Z Disabled control of system clock
I20260620 15:04:50.898654 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43369
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:43369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:04:51.249155 19970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:51.249570 19970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:51.249660 19970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:51.256857 19970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:04:51.256973 19970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:51.257010 19970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:51.257067 19970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:04:51.257112 19970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:04:51.266532 19970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:43369
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:51.267838 19970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:51.269647 19970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:51.281608 19976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:51.281663 19975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:51.282208 19978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:51.282757 19970 server_base.cc:1034] running on GCE node
I20260620 15:04:51.283638 19970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:51.285125 19970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:51.286398 19970 hybrid_clock.cc:648] HybridClock initialized: now 1781967891286318 us; error 56 us; skew 500 ppm
I20260620 15:04:51.286814 19970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:04:51.292874 19970 webserver.cc:460] Webserver started at http://127.2.70.62:34357/ using document root <none> and password file <none>
I20260620 15:04:51.293677 19970 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:51.293805 19970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:51.294150 19970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:51.297153 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4"
format_stamp: "Formatted at 2026-06-20 15:04:51 on dist-test-slave-n5vv"
server_key: "a13d4bcc189198c2a8a0643cb3516ce6"
server_key_iv: "da7b62fb5c1e317d6bc4779242f12563"
server_key_version: "encryptionkey@0"
I20260620 15:04:51.298003 19970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4"
format_stamp: "Formatted at 2026-06-20 15:04:51 on dist-test-slave-n5vv"
server_key: "a13d4bcc189198c2a8a0643cb3516ce6"
server_key_iv: "da7b62fb5c1e317d6bc4779242f12563"
server_key_version: "encryptionkey@0"
I20260620 15:04:51.303179 19970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:04:51.306496 19984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:51.307859 19970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:04:51.308045 19970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4"
format_stamp: "Formatted at 2026-06-20 15:04:51 on dist-test-slave-n5vv"
server_key: "a13d4bcc189198c2a8a0643cb3516ce6"
server_key_iv: "da7b62fb5c1e317d6bc4779242f12563"
server_key_version: "encryptionkey@0"
I20260620 15:04:51.308255 19970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:51.326330 19970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:51.327222 19970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:51.327519 19970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:51.346079 19970 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:43369
I20260620 15:04:51.346113 20035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:43369 every 8 connection(s)
I20260620 15:04:51.347452 19970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:04:51.351173 20036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:51.351366 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 19970
I20260620 15:04:51.351545 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:04:51.352006 2328 external_mini_cluster.cc:1428] Setting key 8b1761e632bbb2e8828a4e16997b46cc
I20260620 15:04:51.365922 20036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4: Bootstrap starting.
I20260620 15:04:51.368659 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:51.353939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52068 (local address 127.2.70.62:43369)
0620 15:04:51.355054 (+ 1115us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:51.355069 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:51.355120 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:04:51.357924 (+ 2804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:51.357953 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:51.358587 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:51.359051 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:51.359721 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:51.361661 (+ 1940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:51.363090 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:51.364195 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:51.364377 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:51.365707 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:51.365718 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:51.365737 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:51.366473 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:51.366581 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:51.366612 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:51.366720 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:51.366969 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 15:04:51.367295 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":321,"threads_started":1}
I20260620 15:04:51.369889 20036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:51.371251 20036 log.cc:824] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:51.374140 20036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4: No bootstrap required, opened a new log
I20260620 15:04:51.381881 20036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43369 } }
I20260620 15:04:51.382244 20036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:51.382320 20036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb1caf695ac49eabdd1eebf7c4fd1a4, State: Initialized, Role: FOLLOWER
I20260620 15:04:51.382889 20036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43369 } }
I20260620 15:04:51.383178 20036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:04:51.383345 20036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:04:51.383553 20036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:51.385845 20036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43369 } }
I20260620 15:04:51.386345 20036 leader_election.cc:302] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fb1caf695ac49eabdd1eebf7c4fd1a4; no voters:
I20260620 15:04:51.387005 20036 leader_election.cc:288] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:04:51.387182 20041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:04:51.387980 20041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [term 1 LEADER]: Becoming Leader. State: Replica: 2fb1caf695ac49eabdd1eebf7c4fd1a4, State: Running, Role: LEADER
I20260620 15:04:51.388633 20041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43369 } }
I20260620 15:04:51.389077 20036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:04:51.392156 20043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fb1caf695ac49eabdd1eebf7c4fd1a4. Latest consensus state: current_term: 1 leader_uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43369 } } }
I20260620 15:04:51.392623 20043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:51.392510 20042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb1caf695ac49eabdd1eebf7c4fd1a4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 43369 } } }
I20260620 15:04:51.393028 20042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4 [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:51.393792 20050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:04:51.399735 20050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:04:51.409597 20050 catalog_manager.cc:1347] Generated new cluster ID: 27463c8779094a288055c2a711b8039f
I20260620 15:04:51.409759 20050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:04:51.418802 20050 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:04:51.420622 20050 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:04:51.426690 20050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4: Generated new TSK 0
I20260620 15:04:51.427493 20050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:04:51.434298 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43369
--builtin_ntp_servers=127.2.70.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:04:51.766134 20060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:51.766453 20060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:51.766618 20060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:51.773941 20060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:51.774076 20060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:51.774251 20060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:04:51.783315 20060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:51.784624 20060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:51.786469 20060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:51.800016 20068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:51.799971 20065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:51.800055 20066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:51.800935 20060 server_base.cc:1034] running on GCE node
I20260620 15:04:51.801611 20060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:51.802484 20060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:51.803690 20060 hybrid_clock.cc:648] HybridClock initialized: now 1781967891803647 us; error 29 us; skew 500 ppm
I20260620 15:04:51.804034 20060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:04:51.806268 20060 webserver.cc:460] Webserver started at http://127.2.70.1:35641/ using document root <none> and password file <none>
I20260620 15:04:51.806963 20060 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:51.807078 20060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:51.807416 20060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:51.810299 20060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "4ee1ca23924f455781b7d1b592981c2f"
format_stamp: "Formatted at 2026-06-20 15:04:51 on dist-test-slave-n5vv"
server_key: "eff2a8db5c8191ff875a179d0d6b1874"
server_key_iv: "48169c5fb2fe3e14606cd90a0724b1f3"
server_key_version: "encryptionkey@0"
I20260620 15:04:51.811080 20060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "4ee1ca23924f455781b7d1b592981c2f"
format_stamp: "Formatted at 2026-06-20 15:04:51 on dist-test-slave-n5vv"
server_key: "eff2a8db5c8191ff875a179d0d6b1874"
server_key_iv: "48169c5fb2fe3e14606cd90a0724b1f3"
server_key_version: "encryptionkey@0"
I20260620 15:04:51.816337 20060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:04:51.819814 20074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:51.821095 20060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 15:04:51.821290 20060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "4ee1ca23924f455781b7d1b592981c2f"
format_stamp: "Formatted at 2026-06-20 15:04:51 on dist-test-slave-n5vv"
server_key: "eff2a8db5c8191ff875a179d0d6b1874"
server_key_iv: "48169c5fb2fe3e14606cd90a0724b1f3"
server_key_version: "encryptionkey@0"
I20260620 15:04:51.821529 20060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:51.844408 20060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:51.845295 20060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:51.845669 20060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:51.846652 20060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:51.848151 20060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:51.848291 20060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:51.848425 20060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:51.848517 20060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:51.882942 20060 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:35397
I20260620 15:04:51.882992 20186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:35397 every 8 connection(s)
I20260620 15:04:51.884282 20060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:04:51.891224 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 20060
I20260620 15:04:51.891458 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:04:51.892076 2328 external_mini_cluster.cc:1428] Setting key c5d882f176abbbd5ad703db72741325e
I20260620 15:04:51.895205 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43369
--builtin_ntp_servers=127.2.70.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:04:51.898650 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:51.886830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:56259 (local address 127.2.70.62:43369)
0620 15:04:51.887310 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:51.887316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:51.888381 (+ 1065us) server_negotiation.cc:407] Connection header received
0620 15:04:51.890400 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:51.890407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:51.890505 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:51.890630 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:51.892237 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:51.893252 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:51.895728 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:51.896111 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:51.896210 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:51.897901 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:51.897906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:51.897909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:51.897927 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:51.897975 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:51.897988 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:51.898054 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:51.898314 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 15:04:51.898390 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":256,"threads_started":1}
I20260620 15:04:51.900142 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:51.887257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43369 (local address 127.2.70.1:56259)
0620 15:04:51.888212 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:51.888249 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:51.890063 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:51.890931 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:51.890945 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:51.891089 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:51.892015 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:51.892041 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:51.893422 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:51.893430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:51.895587 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:51.895604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:51.896352 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:51.896361 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:51.896708 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:51.897501 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:04:51.897508 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:51.897580 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:04:51.897700 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:51.898085 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:51.898098 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:51.898109 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 15:04:51.898289 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 15:04:51.898729 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":327,"threads_started":1}
I20260620 15:04:51.901882 20187 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43369
I20260620 15:04:51.902200 20187 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:51.902915 20187 heartbeater.cc:511] Master 127.2.70.62:43369 requested a full tablet report, sending...
I20260620 15:04:51.905052 20001 ts_manager.cc:194] Registered new tserver with Master: 4ee1ca23924f455781b7d1b592981c2f (127.2.70.1:35397)
I20260620 15:04:51.906582 20001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.1:56259
W20260620 15:04:52.225766 20191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:52.226082 20191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:52.226246 20191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:52.234052 20191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:52.234165 20191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:52.234400 20191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:04:52.244131 20191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:52.245527 20191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:52.247304 20191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:52.260334 20197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:52.260401 20199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:52.260365 20196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:52.261552 20191 server_base.cc:1034] running on GCE node
I20260620 15:04:52.262198 20191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:52.263188 20191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:52.264425 20191 hybrid_clock.cc:648] HybridClock initialized: now 1781967892264373 us; error 64 us; skew 500 ppm
I20260620 15:04:52.264744 20191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:04:52.267088 20191 webserver.cc:460] Webserver started at http://127.2.70.2:44623/ using document root <none> and password file <none>
I20260620 15:04:52.267778 20191 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:52.267910 20191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:52.268231 20191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:52.271189 20191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "4fdf64e9bba347e893e1586c61ab08ae"
format_stamp: "Formatted at 2026-06-20 15:04:52 on dist-test-slave-n5vv"
server_key: "6d0840510b100c4587e185b8574348da"
server_key_iv: "d4959576ff8a68cd89a800619d7f41c2"
server_key_version: "encryptionkey@0"
I20260620 15:04:52.271976 20191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "4fdf64e9bba347e893e1586c61ab08ae"
format_stamp: "Formatted at 2026-06-20 15:04:52 on dist-test-slave-n5vv"
server_key: "6d0840510b100c4587e185b8574348da"
server_key_iv: "d4959576ff8a68cd89a800619d7f41c2"
server_key_version: "encryptionkey@0"
I20260620 15:04:52.276979 20191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:04:52.280767 20205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:52.282152 20191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:52.282341 20191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "4fdf64e9bba347e893e1586c61ab08ae"
format_stamp: "Formatted at 2026-06-20 15:04:52 on dist-test-slave-n5vv"
server_key: "6d0840510b100c4587e185b8574348da"
server_key_iv: "d4959576ff8a68cd89a800619d7f41c2"
server_key_version: "encryptionkey@0"
I20260620 15:04:52.282575 20191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:52.305785 20191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:52.306638 20191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:52.306917 20191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:52.307858 20191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:52.309329 20191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:52.309453 20191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:52.309597 20191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:52.309687 20191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:52.342638 20191 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:40997
I20260620 15:04:52.342723 20317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:40997 every 8 connection(s)
I20260620 15:04:52.343920 20191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:04:52.351830 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 20191
I20260620 15:04:52.352020 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:04:52.352643 2328 external_mini_cluster.cc:1428] Setting key 47226a7b213a266fadcbaf927d6962f0
I20260620 15:04:52.355369 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43369
--builtin_ntp_servers=127.2.70.20:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:04:52.359128 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.346667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:38681 (local address 127.2.70.62:43369)
0620 15:04:52.346888 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:52.346894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:52.348384 (+ 1490us) server_negotiation.cc:407] Connection header received
0620 15:04:52.350913 (+ 2529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:52.350918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:52.350987 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:52.351076 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:52.352375 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.353433 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.355715 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.356456 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.356602 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.358191 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:52.358197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:52.358203 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:52.358227 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:52.358270 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:52.358291 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:52.358383 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:52.358624 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 15:04:52.358736 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:04:52.360221 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.347225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43369 (local address 127.2.70.2:38681)
0620 15:04:52.348216 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:52.348258 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:52.350601 (+ 2343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:52.351338 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:52.351349 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:52.351468 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:52.352186 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.352209 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.353622 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.353628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.355574 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.355588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.356675 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.356681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.356962 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.357826 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:04:52.357830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:52.357880 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:04:52.357984 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:52.358392 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:52.358402 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:52.358416 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 15:04:52.358595 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 15:04:52.358968 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":274,"threads_started":1}
I20260620 15:04:52.361752 20318 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43369
I20260620 15:04:52.362035 20318 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:52.362839 20318 heartbeater.cc:511] Master 127.2.70.62:43369 requested a full tablet report, sending...
I20260620 15:04:52.364400 20001 ts_manager.cc:194] Registered new tserver with Master: 4fdf64e9bba347e893e1586c61ab08ae (127.2.70.2:40997)
I20260620 15:04:52.365703 20001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.2:38681
W20260620 15:04:52.688803 20321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:52.689090 20321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:52.689177 20321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:52.696818 20321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:52.696924 20321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:52.697031 20321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:04:52.706043 20321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:43369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:52.707260 20321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:52.708954 20321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:52.721819 20326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:52.721851 20327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:52.721758 20329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:52.722707 20321 server_base.cc:1034] running on GCE node
I20260620 15:04:52.723287 20321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:52.724265 20321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:52.725526 20321 hybrid_clock.cc:648] HybridClock initialized: now 1781967892725447 us; error 66 us; skew 500 ppm
I20260620 15:04:52.725862 20321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:04:52.728070 20321 webserver.cc:460] Webserver started at http://127.2.70.3:45413/ using document root <none> and password file <none>
I20260620 15:04:52.728772 20321 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:52.728897 20321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:52.729229 20321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:52.732097 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c"
format_stamp: "Formatted at 2026-06-20 15:04:52 on dist-test-slave-n5vv"
server_key: "1f6f576f555db8d45e14fe504a8d6d48"
server_key_iv: "7b42d2381c6ae87b18d0282524c5bf33"
server_key_version: "encryptionkey@0"
I20260620 15:04:52.732897 20321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c"
format_stamp: "Formatted at 2026-06-20 15:04:52 on dist-test-slave-n5vv"
server_key: "1f6f576f555db8d45e14fe504a8d6d48"
server_key_iv: "7b42d2381c6ae87b18d0282524c5bf33"
server_key_version: "encryptionkey@0"
I20260620 15:04:52.737970 20321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:04:52.741276 20335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:52.742614 20321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:04:52.742785 20321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c"
format_stamp: "Formatted at 2026-06-20 15:04:52 on dist-test-slave-n5vv"
server_key: "1f6f576f555db8d45e14fe504a8d6d48"
server_key_iv: "7b42d2381c6ae87b18d0282524c5bf33"
server_key_version: "encryptionkey@0"
I20260620 15:04:52.743009 20321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:52.756222 20321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:52.757051 20321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:52.757331 20321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:52.758401 20321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:52.759886 20321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:52.760007 20321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:52.760145 20321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:52.760234 20321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:52.793592 20321 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:33729
I20260620 15:04:52.793685 20447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:33729 every 8 connection(s)
I20260620 15:04:52.794917 20321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:04:52.796734 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 20321
I20260620 15:04:52.796936 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:04:52.797436 2328 external_mini_cluster.cc:1428] Setting key 35457d457f7792fe743ed47a60a74762
I20260620 15:04:52.807564 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.797824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:54559 (local address 127.2.70.62:43369)
0620 15:04:52.798061 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:52.798067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:52.799632 (+ 1565us) server_negotiation.cc:407] Connection header received
0620 15:04:52.801421 (+ 1789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:52.801425 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:52.801563 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:52.801659 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:52.802730 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.803485 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.804982 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.805435 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.805630 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.806849 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:52.806855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:52.806858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:52.806877 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:52.806908 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:52.806920 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:52.806995 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:52.807185 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 15:04:52.807263 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 15:04:52.808501 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.798430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:43369 (local address 127.2.70.3:54559)
0620 15:04:52.799477 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:52.799524 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:52.801136 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:52.801868 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:52.801876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:52.801982 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:52.802566 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.802582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.803613 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.803617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.804830 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.804841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.805701 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.805705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.805898 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.806537 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:04:52.806539 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:52.806577 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:04:52.806653 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:52.807007 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:52.807014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:52.807020 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:04:52.807158 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 15:04:52.807467 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":340,"threads_started":1}
I20260620 15:04:52.809991 20448 heartbeater.cc:347] Connected to a master server at 127.2.70.62:43369
I20260620 15:04:52.810256 20448 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:52.811254 20448 heartbeater.cc:511] Master 127.2.70.62:43369 requested a full tablet report, sending...
I20260620 15:04:52.812954 20001 ts_manager.cc:194] Registered new tserver with Master: 843e9ec690af40b4a0bca9ab3ffcfd1c (127.2.70.3:33729)
I20260620 15:04:52.813920 20001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.3:54559
I20260620 15:04:52.825101 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:04:52.835197 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.829245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37376 (local address 127.2.70.62:43369)
0620 15:04:52.829454 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:52.829464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:52.829959 (+ 495us) server_negotiation.cc:407] Connection header received
0620 15:04:52.830097 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:52.830101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:52.830183 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:52.830286 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:52.830762 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.831446 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.832777 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.833188 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.833281 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.834423 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:52.834427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:52.834431 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:52.834449 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:52.834481 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:52.834492 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:52.834560 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:52.834678 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 15:04:52.834750 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:04:52.839066 20001 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37376:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:04:52.845985 20001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:04:52.874092 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.860111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.0.0.1:43278)
0620 15:04:52.860916 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:52.860935 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:52.861060 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:52.866267 (+ 5207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:52.866274 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:52.866305 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:52.866679 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.866693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.868738 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.868745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.871037 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.871053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.872975 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.872982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.873191 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.873250 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 15:04:52.873377 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 15:04:52.873632 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":3,"thread_start_us":5617,"threads_started":1}
I20260620 15:04:52.874264 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.861224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.2.70.3:33729)
0620 15:04:52.865067 (+ 3843us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:52.865076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:52.866262 (+ 1186us) server_negotiation.cc:407] Connection header received
0620 15:04:52.866364 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:52.866375 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:52.866626 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:52.866836 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:52.867699 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.869325 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.871711 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.873113 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.873251 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.873580 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:52.873734 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 15:04:52.873896 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3659,"thread_start_us":3971,"threads_started":1}
I20260620 15:04:52.874202 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.860343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:33729 (local address 127.0.0.1:50300)
0620 15:04:52.865990 (+ 5647us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:52.866008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:52.866134 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:52.866850 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:52.866858 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:52.866886 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:52.867463 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.867476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.869556 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.869561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.871506 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.871520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.873317 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.873324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.873546 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.873583 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:04:52.873678 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 15:04:52.873806 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5519,"mutex_wait_us":128,"thread_start_us":3769,"threads_started":1}
I20260620 15:04:52.875010 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.858982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.2.70.2:40997)
0620 15:04:52.866352 (+ 7370us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:52.866361 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:52.866397 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:04:52.866485 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:52.866490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:52.866662 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:52.867464 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:52.867730 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.869324 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.871928 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.873377 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.874190 (+ 813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.874455 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:52.874522 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 15:04:52.874686 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7227,"thread_start_us":7116,"threads_started":1}
I20260620 15:04:52.874775 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.858715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:40997 (local address 127.0.0.1:35032)
0620 15:04:52.859451 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:52.859483 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:52.859618 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:52.866882 (+ 7264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:52.866886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:52.866904 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:52.867484 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.867501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.869475 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.869501 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.871731 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:52.871744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.873621 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.873627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:52.873800 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.873835 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 15:04:52.873899 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 15:04:52.874034 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":162,"spinlock_wait_cycles":2304,"thread_start_us":360,"threads_started":1}
I20260620 15:04:52.875339 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:52.860750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43278 (local address 127.2.70.1:35397)
0620 15:04:52.865544 (+ 4794us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:52.865552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:52.865585 (+ 33us) server_negotiation.cc:407] Connection header received
0620 15:04:52.865702 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:52.865713 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:52.866018 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:52.866861 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:52.867128 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.868595 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.871235 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:52.872738 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:52.874577 (+ 1839us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:52.874732 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:52.874782 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 15:04:52.874961 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4602,"thread_start_us":4656,"threads_started":1}
I20260620 15:04:52.879112 20383 tablet_service.cc:1467] Processing CreateTablet for tablet ab95e3bb235a4b608a0ae91b57dd3474 (DEFAULT_TABLE table=test-table [id=b3a245110e164ac7a816d8b16a6fc829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:52.879110 20122 tablet_service.cc:1467] Processing CreateTablet for tablet ab95e3bb235a4b608a0ae91b57dd3474 (DEFAULT_TABLE table=test-table [id=b3a245110e164ac7a816d8b16a6fc829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:52.879110 20253 tablet_service.cc:1467] Processing CreateTablet for tablet ab95e3bb235a4b608a0ae91b57dd3474 (DEFAULT_TABLE table=test-table [id=b3a245110e164ac7a816d8b16a6fc829]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:52.880434 20383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab95e3bb235a4b608a0ae91b57dd3474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:52.880481 20253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab95e3bb235a4b608a0ae91b57dd3474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:52.880515 20122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab95e3bb235a4b608a0ae91b57dd3474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:52.898270 20466 tablet_bootstrap.cc:492] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Bootstrap starting.
I20260620 15:04:52.900444 20467 tablet_bootstrap.cc:492] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: Bootstrap starting.
I20260620 15:04:52.902431 20466 tablet_bootstrap.cc:654] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:52.902421 20468 tablet_bootstrap.cc:492] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f: Bootstrap starting.
I20260620 15:04:52.904156 20467 tablet_bootstrap.cc:654] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:52.904665 20466 log.cc:824] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:52.905464 20467 log.cc:824] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:52.905826 20468 tablet_bootstrap.cc:654] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:52.906785 20468 log.cc:824] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:52.908205 20466 tablet_bootstrap.cc:492] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: No bootstrap required, opened a new log
I20260620 15:04:52.908586 20466 ts_tablet_manager.cc:1397] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 15:04:52.909633 20467 tablet_bootstrap.cc:492] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: No bootstrap required, opened a new log
I20260620 15:04:52.909911 20468 tablet_bootstrap.cc:492] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f: No bootstrap required, opened a new log
I20260620 15:04:52.909916 20467 ts_tablet_manager.cc:1397] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.002s
I20260620 15:04:52.910190 20187 heartbeater.cc:503] Master 127.2.70.62:43369 was elected leader, sending a full tablet report...
I20260620 15:04:52.910231 20468 ts_tablet_manager.cc:1397] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260620 15:04:52.919515 20467 raft_consensus.cc:348] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:52.919958 20467 raft_consensus.cc:374] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:52.920084 20467 raft_consensus.cc:729] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fdf64e9bba347e893e1586c61ab08ae, State: Initialized, Role: FOLLOWER
I20260620 15:04:52.920351 20466 raft_consensus.cc:348] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:52.920802 20466 raft_consensus.cc:374] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:52.920921 20466 raft_consensus.cc:729] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 843e9ec690af40b4a0bca9ab3ffcfd1c, State: Initialized, Role: FOLLOWER
I20260620 15:04:52.920791 20467 consensus_queue.cc:260] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:52.921142 20468 raft_consensus.cc:348] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:52.921689 20468 raft_consensus.cc:374] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:52.921630 20466 consensus_queue.cc:260] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:52.921833 20468 raft_consensus.cc:729] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee1ca23924f455781b7d1b592981c2f, State: Initialized, Role: FOLLOWER
I20260620 15:04:52.922312 20318 heartbeater.cc:503] Master 127.2.70.62:43369 was elected leader, sending a full tablet report...
I20260620 15:04:52.922547 20468 consensus_queue.cc:260] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:52.922916 20467 ts_tablet_manager.cc:1428] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260620 15:04:52.924669 20448 heartbeater.cc:503] Master 127.2.70.62:43369 was elected leader, sending a full tablet report...
I20260620 15:04:52.925720 20466 ts_tablet_manager.cc:1428] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260620 15:04:52.930864 20468 ts_tablet_manager.cc:1428] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f: Time spent starting tablet: real 0.020s user 0.015s sys 0.003s
I20260620 15:04:53.004237 20472 raft_consensus.cc:482] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:04:53.004578 20472 raft_consensus.cc:504] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:53.006271 20472 leader_election.cc:288] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ee1ca23924f455781b7d1b592981c2f (127.2.70.1:35397), 843e9ec690af40b4a0bca9ab3ffcfd1c (127.2.70.3:33729)
I20260620 15:04:53.015925 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.006822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:36739 (local address 127.2.70.1:35397)
0620 15:04:53.007004 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:53.007011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:53.007920 (+ 909us) server_negotiation.cc:407] Connection header received
0620 15:04:53.008117 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:53.008122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:53.008222 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:53.008368 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:53.009209 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.010349 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.013570 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.014843 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.014976 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:53.015072 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:53.015520 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 15:04:53.015652 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:04:53.015925 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.006798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:59611 (local address 127.2.70.3:33729)
0620 15:04:53.007004 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:53.007011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:53.007923 (+ 912us) server_negotiation.cc:407] Connection header received
0620 15:04:53.008098 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:53.008104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:53.008190 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:53.008303 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:53.009209 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.010359 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.013570 (+ 3211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.014845 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.014976 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:53.015073 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:53.015520 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 15:04:53.015652 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:04:53.016108 20476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.006602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.2.70.2:36739)
0620 15:04:53.007755 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.007780 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:53.007980 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:53.008384 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:53.008390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:53.008449 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:53.009069 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:53.009081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.010522 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.010528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:53.013312 (+ 2784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:53.013329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.015110 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.015116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:53.015300 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:53.015342 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 15:04:53.015447 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 15:04:53.015652 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":975,"mutex_wait_us":288,"spinlock_wait_cycles":72960,"thread_start_us":805,"threads_started":1}
I20260620 15:04:53.016261 20475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.006658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:33729 (local address 127.2.70.2:59611)
0620 15:04:53.007788 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.007804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:53.007948 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:53.008384 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:53.008390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:53.008447 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:53.008997 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:53.009016 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.010521 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.010528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:53.013315 (+ 2787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:53.013329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.015104 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.015111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:53.015358 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:53.015389 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 15:04:53.015465 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 15:04:53.015652 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"mutex_wait_us":207,"spinlock_wait_cycles":13568,"thread_start_us":539,"threads_started":1}
I20260620 15:04:53.017342 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab95e3bb235a4b608a0ae91b57dd3474" candidate_uuid: "4fdf64e9bba347e893e1586c61ab08ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" is_pre_election: true
I20260620 15:04:53.017329 20142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab95e3bb235a4b608a0ae91b57dd3474" candidate_uuid: "4fdf64e9bba347e893e1586c61ab08ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee1ca23924f455781b7d1b592981c2f" is_pre_election: true
I20260620 15:04:53.017935 20142 raft_consensus.cc:2454] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fdf64e9bba347e893e1586c61ab08ae in term 0.
I20260620 15:04:53.017933 20403 raft_consensus.cc:2454] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fdf64e9bba347e893e1586c61ab08ae in term 0.
I20260620 15:04:53.018652 20208 leader_election.cc:302] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4ee1ca23924f455781b7d1b592981c2f, 4fdf64e9bba347e893e1586c61ab08ae, 843e9ec690af40b4a0bca9ab3ffcfd1c; no voters:
I20260620 15:04:53.019340 20472 raft_consensus.cc:2781] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:04:53.019498 20472 raft_consensus.cc:482] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:04:53.019626 20472 raft_consensus.cc:3037] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:53.022008 20472 raft_consensus.cc:504] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:53.022595 20472 leader_election.cc:288] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [CANDIDATE]: Term 1 election: Requested vote from peers 4ee1ca23924f455781b7d1b592981c2f (127.2.70.1:35397), 843e9ec690af40b4a0bca9ab3ffcfd1c (127.2.70.3:33729)
I20260620 15:04:53.023223 20142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab95e3bb235a4b608a0ae91b57dd3474" candidate_uuid: "4fdf64e9bba347e893e1586c61ab08ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee1ca23924f455781b7d1b592981c2f"
I20260620 15:04:53.023311 20403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab95e3bb235a4b608a0ae91b57dd3474" candidate_uuid: "4fdf64e9bba347e893e1586c61ab08ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c"
I20260620 15:04:53.023450 20142 raft_consensus.cc:3037] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:53.023501 20403 raft_consensus.cc:3037] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:53.026801 20403 raft_consensus.cc:2454] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fdf64e9bba347e893e1586c61ab08ae in term 1.
I20260620 15:04:53.026801 20142 raft_consensus.cc:2454] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fdf64e9bba347e893e1586c61ab08ae in term 1.
I20260620 15:04:53.027237 20208 leader_election.cc:302] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fdf64e9bba347e893e1586c61ab08ae, 843e9ec690af40b4a0bca9ab3ffcfd1c; no voters:
I20260620 15:04:53.027549 20472 raft_consensus.cc:2781] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:04:53.028116 20472 raft_consensus.cc:686] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 1 LEADER]: Becoming Leader. State: Replica: 4fdf64e9bba347e893e1586c61ab08ae, State: Running, Role: LEADER
I20260620 15:04:53.028599 20472 consensus_queue.cc:237] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } }
I20260620 15:04:53.033382 20001 catalog_manager.cc:5495] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fdf64e9bba347e893e1586c61ab08ae (127.2.70.2). New cstate: current_term: 1 leader_uuid: "4fdf64e9bba347e893e1586c61ab08ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fdf64e9bba347e893e1586c61ab08ae" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 } health_report { overall_health: UNKNOWN } } }
W20260620 15:04:53.046713 20449 tablet.cc:2369] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:04:53.068516 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.063246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.2.70.2:40997)
0620 15:04:53.063471 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:53.063475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:53.063490 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:04:53.063572 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:53.063575 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:53.063657 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:53.063806 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:04:53.064222 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.064893 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.066398 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.066810 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.066935 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:53.067415 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:04:53.067916 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:04:53.068076 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:53.068172 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 15:04:53.068249 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:04:53.076608 20142 raft_consensus.cc:1261] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 1 FOLLOWER]: Refusing update from remote peer 4fdf64e9bba347e893e1586c61ab08ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:04:53.076606 20403 raft_consensus.cc:1261] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 1 FOLLOWER]: Refusing update from remote peer 4fdf64e9bba347e893e1586c61ab08ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:04:53.077579 20477 consensus_queue.cc:1035] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "843e9ec690af40b4a0bca9ab3ffcfd1c" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 33729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:04:53.077912 20472 consensus_queue.cc:1035] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee1ca23924f455781b7d1b592981c2f" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 35397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:04:53.103420 20000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:04:53.103718 20000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37376:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:04:53.107194 20000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fb1caf695ac49eabdd1eebf7c4fd1a4: Sending DeleteTablet for 3 replicas of tablet ab95e3bb235a4b608a0ae91b57dd3474
I20260620 15:04:53.108318 20253 tablet_service.cc:1514] Processing DeleteTablet for tablet ab95e3bb235a4b608a0ae91b57dd3474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:53 UTC) from {username='slave'} at 127.0.0.1:35032
I20260620 15:04:53.108485 20122 tablet_service.cc:1514] Processing DeleteTablet for tablet ab95e3bb235a4b608a0ae91b57dd3474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:53 UTC) from {username='slave'} at 127.0.0.1:43278
I20260620 15:04:53.108690 20383 tablet_service.cc:1514] Processing DeleteTablet for tablet ab95e3bb235a4b608a0ae91b57dd3474 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:53 UTC) from {username='slave'} at 127.0.0.1:50300
I20260620 15:04:53.110790 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 20060
I20260620 15:04:53.124740 20491 tablet_replica.cc:330] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f: stopping tablet replica
I20260620 15:04:53.126012 20492 tablet_replica.cc:330] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: stopping tablet replica
I20260620 15:04:53.126550 20490 tablet_replica.cc:330] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: stopping tablet replica
I20260620 15:04:53.127022 20490 raft_consensus.cc:2229] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:04:53.127334 20492 raft_consensus.cc:2229] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:04:53.127064 20491 raft_consensus.cc:2229] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:04:53.127738 20490 raft_consensus.cc:2258] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:53.128631 20491 raft_consensus.cc:2258] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:53.128834 20492 raft_consensus.cc:2258] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:53.130005 20490 ts_tablet_manager.cc:1905] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:53.131795 20492 ts_tablet_manager.cc:1905] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:53.132973 20491 ts_tablet_manager.cc:1905] T ab95e3bb235a4b608a0ae91b57dd3474 P 4ee1ca23924f455781b7d1b592981c2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:53.134883 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.111937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.2.70.1:35397)
0620 15:04:53.114617 (+ 2680us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:53.114627 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:53.125914 (+ 11287us) server_negotiation.cc:407] Connection header received
0620 15:04:53.126178 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:53.126183 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:53.126296 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:53.126408 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:53.127071 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.128237 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.130578 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.131213 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.131343 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:53.132753 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:53.132763 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:53.132778 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:53.133573 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:53.133633 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:53.133657 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:53.133763 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:53.134425 (+ 662us) server_negotiation.cc:299] Negotiation successful
0620 15:04:53.134534 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2498}
I20260620 15:04:53.135682 20182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:53.137678 20490 ts_tablet_manager.cc:1918] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:53.137920 20490 log.cc:1196] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/ab95e3bb235a4b608a0ae91b57dd3474
I20260620 15:04:53.138566 20490 ts_tablet_manager.cc:1939] T ab95e3bb235a4b608a0ae91b57dd3474 P 4fdf64e9bba347e893e1586c61ab08ae: Deleting consensus metadata
I20260620 15:04:53.140462 19985 catalog_manager.cc:4841] TS 4fdf64e9bba347e893e1586c61ab08ae (127.2.70.2:40997): tablet ab95e3bb235a4b608a0ae91b57dd3474 (table test-table [id=b3a245110e164ac7a816d8b16a6fc829]) successfully deleted
I20260620 15:04:53.142369 20492 ts_tablet_manager.cc:1918] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:53.142643 20492 log.cc:1196] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/ab95e3bb235a4b608a0ae91b57dd3474
I20260620 15:04:53.143329 20492 ts_tablet_manager.cc:1939] T ab95e3bb235a4b608a0ae91b57dd3474 P 843e9ec690af40b4a0bca9ab3ffcfd1c: Deleting consensus metadata
I20260620 15:04:53.145208 19987 catalog_manager.cc:4841] TS 843e9ec690af40b4a0bca9ab3ffcfd1c (127.2.70.3:33729): tablet ab95e3bb235a4b608a0ae91b57dd3474 (table test-table [id=b3a245110e164ac7a816d8b16a6fc829]) successfully deleted
I20260620 15:04:53.242830 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 20060
W20260620 15:04:53.261293 19985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.1:35397 (error 108)
I20260620 15:04:53.261417 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 20191
I20260620 15:04:53.263352 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.262866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.0.0.1:43300)
0620 15:04:53.263050 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.263176 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 15:04:53.264567 19985 catalog_manager.cc:4584] TS 4ee1ca23924f455781b7d1b592981c2f (127.2.70.1:35397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ab95e3bb235a4b608a0ae91b57dd3474: Network error: Client connection negotiation failed: client connection to 127.2.70.1:35397: connect: Connection refused (error 111)
I20260620 15:04:53.269013 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.262518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.2.70.2:40997)
0620 15:04:53.262710 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:53.262716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:53.262735 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:04:53.262803 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:53.262813 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:53.262938 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:53.263056 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:53.263579 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.264326 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.265882 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.266306 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.266474 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:53.267623 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:53.267632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:53.267646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:53.268244 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:53.268291 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:53.268310 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:53.268438 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:53.268572 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 15:04:53.268673 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:04:53.269716 20313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:53.318315 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.317844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.0.0.1:43306)
0620 15:04:53.318057 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.318139 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:04:53.319339 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.318994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.0.0.1:43308)
0620 15:04:53.319140 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.319226 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:04:53.379666 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.379207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.0.0.1:43320)
0620 15:04:53.379399 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.379501 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:04:53.380647 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.380320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.0.0.1:43322)
0620 15:04:53.380463 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.380540 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 15:04:53.382795 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 20191
I20260620 15:04:53.401809 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 20321
I20260620 15:04:53.408337 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.402552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.2.70.3:33729)
0620 15:04:53.402731 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:53.402736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:53.402757 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:04:53.402965 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:53.402969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:53.403054 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:53.403150 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:53.403507 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.404270 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.405570 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:53.405985 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:53.406083 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:53.407140 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:53.407149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:53.407164 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:53.407738 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:53.407787 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:53.407806 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:53.407899 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:53.408003 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 15:04:53.408093 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:04:53.409024 20443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:53.461468 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.460827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.0.0.1:43328)
0620 15:04:53.461102 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.461250 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260620 15:04:53.462756 20462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:53.462294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:35397 (local address 127.0.0.1:43344)
0620 15:04:53.462518 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:53.462605 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:35397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 15:04:53.501088 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 20321
I20260620 15:04:53.518196 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 19970
I20260620 15:04:53.519371 20031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:53.629053 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 19970
2026-06-20T15:04:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2809 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:04:53 dist-test-slave-n5vv krb5kdc[20501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:04:53 dist-test-slave-n5vv krb5kdc[20501](info): set up 2 sockets
Jun 20 15:04:53 dist-test-slave-n5vv krb5kdc[20501](info): commencing operation
krb5kdc: starting...
W20260620 15:04:54.749670 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:04:54 dist-test-slave-n5vv krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:04:55.000550 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42099
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:42099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:55.333753 20514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:55.334039 20514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:55.334131 20514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:55.341293 20514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:04:55.341409 20514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:55.341456 20514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:55.341586 20514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:04:55.341635 20514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:04:55.351655 20514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:42099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:55.352921 20514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:55.354375 20514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:55.366319 20520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:55.366348 20519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:55.366390 20522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:55.367616 20514 server_base.cc:1034] running on GCE node
I20260620 15:04:55.368418 20514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:55.370008 20514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:55.371248 20514 hybrid_clock.cc:648] HybridClock initialized: now 1781967895371201 us; error 54 us; skew 500 ppm
I20260620 15:04:55.371637 20514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:55 dist-test-slave-n5vv krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:55.374961 20514 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:04:55.376449 20514 webserver.cc:460] Webserver started at http://127.2.70.62:37889/ using document root <none> and password file <none>
I20260620 15:04:55.377166 20514 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:55.377336 20514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:55.377697 20514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:55.380751 20514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "b272517ba3aa467fa197b1b66ea7b731"
format_stamp: "Formatted at 2026-06-20 15:04:55 on dist-test-slave-n5vv"
server_key: "a29284cec7a4f64b7c9df126f7f5c150"
server_key_iv: "0e35bb77456b398129df0e108eddf5eb"
server_key_version: "encryptionkey@0"
I20260620 15:04:55.381604 20514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "b272517ba3aa467fa197b1b66ea7b731"
format_stamp: "Formatted at 2026-06-20 15:04:55 on dist-test-slave-n5vv"
server_key: "a29284cec7a4f64b7c9df126f7f5c150"
server_key_iv: "0e35bb77456b398129df0e108eddf5eb"
server_key_version: "encryptionkey@0"
I20260620 15:04:55.386921 20514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:04:55.390511 20529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:55.391867 20514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:04:55.392066 20514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "b272517ba3aa467fa197b1b66ea7b731"
format_stamp: "Formatted at 2026-06-20 15:04:55 on dist-test-slave-n5vv"
server_key: "a29284cec7a4f64b7c9df126f7f5c150"
server_key_iv: "0e35bb77456b398129df0e108eddf5eb"
server_key_version: "encryptionkey@0"
I20260620 15:04:55.392279 20514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:55.406379 20514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:55.410595 20514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:55.410887 20514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:55.429668 20514 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:42099
I20260620 15:04:55.429674 20580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:42099 every 8 connection(s)
I20260620 15:04:55.431113 20514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:04:55.434798 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 20514
I20260620 15:04:55.434917 20581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:55.434971 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:04:55.435415 2328 external_mini_cluster.cc:1428] Setting key 88b8aee4ed8edc6156b7db0cdddfeb7a
I20260620 15:04:55.449326 20581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731: Bootstrap starting.
Jun 20 15:04:55 dist-test-slave-n5vv krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:55.453222 20581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:55.454586 20581 log.cc:824] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:55.458371 20581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731: No bootstrap required, opened a new log
I20260620 15:04:55.460054 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:55.437158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59546 (local address 127.2.70.62:42099)
0620 15:04:55.438004 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:55.438019 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:55.438070 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:04:55.439957 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:55.439987 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:55.440526 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:55.440964 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:55.442363 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:55.444300 (+ 1937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:55.445674 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:55.446788 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:55.446980 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:55.451019 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:55.451030 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:55.451049 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:55.451096 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:55.455014 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:55.456121 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:55.456130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:55.456139 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:55.456341 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:55.456727 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:55.456731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:55.456734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:55.457452 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:55.457763 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:55.458188 (+ 425us) server_negotiation.cc:299] Negotiation successful
0620 15:04:55.458655 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":367,"threads_started":1}
I20260620 15:04:55.466917 20581 raft_consensus.cc:348] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272517ba3aa467fa197b1b66ea7b731" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42099 } }
I20260620 15:04:55.467254 20581 raft_consensus.cc:374] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:55.467383 20581 raft_consensus.cc:729] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b272517ba3aa467fa197b1b66ea7b731, State: Initialized, Role: FOLLOWER
I20260620 15:04:55.467945 20581 consensus_queue.cc:260] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272517ba3aa467fa197b1b66ea7b731" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42099 } }
I20260620 15:04:55.468176 20581 raft_consensus.cc:388] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:04:55.468312 20581 raft_consensus.cc:482] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:04:55.468488 20581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:55.471002 20581 raft_consensus.cc:504] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272517ba3aa467fa197b1b66ea7b731" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42099 } }
I20260620 15:04:55.471518 20581 leader_election.cc:302] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b272517ba3aa467fa197b1b66ea7b731; no voters:
I20260620 15:04:55.472136 20581 leader_election.cc:288] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:04:55.472296 20586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:04:55.473178 20586 raft_consensus.cc:686] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [term 1 LEADER]: Becoming Leader. State: Replica: b272517ba3aa467fa197b1b66ea7b731, State: Running, Role: LEADER
I20260620 15:04:55.473762 20586 consensus_queue.cc:237] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272517ba3aa467fa197b1b66ea7b731" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42099 } }
I20260620 15:04:55.474253 20581 sys_catalog.cc:564] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:04:55.476766 20587 sys_catalog.cc:455] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b272517ba3aa467fa197b1b66ea7b731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272517ba3aa467fa197b1b66ea7b731" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42099 } } }
I20260620 15:04:55.476743 20588 sys_catalog.cc:455] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b272517ba3aa467fa197b1b66ea7b731. Latest consensus state: current_term: 1 leader_uuid: "b272517ba3aa467fa197b1b66ea7b731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b272517ba3aa467fa197b1b66ea7b731" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42099 } } }
I20260620 15:04:55.477288 20587 sys_catalog.cc:458] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:55.477298 20588 sys_catalog.cc:458] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731 [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:55.482152 20593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:04:55.487428 20593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:04:55.497092 20593 catalog_manager.cc:1347] Generated new cluster ID: 3551edb360d749348c703c6ba30d8a6a
I20260620 15:04:55.497409 20593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:04:55.515964 20593 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:04:55.517407 20593 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:04:55.524106 20593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731: Generated new TSK 0
I20260620 15:04:55.524974 20593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:04:55.586668 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42099
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:55.925002 20607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:55.925380 20607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:55.925596 20607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:55.932816 20607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:55.932971 20607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:55.933163 20607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:04:55.943028 20607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:55.944422 20607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:55.945871 20607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:55.960353 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:55.960414 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:55.960439 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:55.963124 20607 server_base.cc:1034] running on GCE node
I20260620 15:04:55.963836 20607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:55.964944 20607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:55.966199 20607 hybrid_clock.cc:648] HybridClock initialized: now 1781967895966144 us; error 47 us; skew 500 ppm
I20260620 15:04:55.966552 20607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:55 dist-test-slave-n5vv krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:55.969897 20607 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:04:55.971484 20607 webserver.cc:460] Webserver started at http://127.2.70.1:37533/ using document root <none> and password file <none>
I20260620 15:04:55.972215 20607 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:55.972357 20607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:55.972663 20607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:55.975852 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "513e77ddaf3e4252b7b8b124e7e91c08"
format_stamp: "Formatted at 2026-06-20 15:04:55 on dist-test-slave-n5vv"
server_key: "15dfee6befda4adb784ea94eafc05484"
server_key_iv: "9cda23b5df0fe694ac8345145690c0ec"
server_key_version: "encryptionkey@0"
I20260620 15:04:55.976610 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "513e77ddaf3e4252b7b8b124e7e91c08"
format_stamp: "Formatted at 2026-06-20 15:04:55 on dist-test-slave-n5vv"
server_key: "15dfee6befda4adb784ea94eafc05484"
server_key_iv: "9cda23b5df0fe694ac8345145690c0ec"
server_key_version: "encryptionkey@0"
I20260620 15:04:55.981861 20607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260620 15:04:55.985314 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:55.986702 20607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:55.986889 20607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "513e77ddaf3e4252b7b8b124e7e91c08"
format_stamp: "Formatted at 2026-06-20 15:04:55 on dist-test-slave-n5vv"
server_key: "15dfee6befda4adb784ea94eafc05484"
server_key_iv: "9cda23b5df0fe694ac8345145690c0ec"
server_key_version: "encryptionkey@0"
I20260620 15:04:55.987092 20607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:56.004410 20607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:56.008637 20607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:56.008955 20607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:56.009968 20607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:56.011318 20607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:56.011435 20607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:56.011591 20607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:56.011664 20607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:56.045642 20607 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:40111
I20260620 15:04:56.045689 20734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:40111 every 8 connection(s)
I20260620 15:04:56.046900 20607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:04:56.054976 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 20607
I20260620 15:04:56.055157 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:04:56.055606 2328 external_mini_cluster.cc:1428] Setting key 3ff5c441c5f060f15264836485ea7eae
Jun 20 15:04:56 dist-test-slave-n5vv krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967895, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:56.068821 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:56.049579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:33785 (local address 127.2.70.62:42099)
0620 15:04:56.050020 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:56.050025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:56.051001 (+ 976us) server_negotiation.cc:407] Connection header received
0620 15:04:56.052901 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:56.052907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:56.053011 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:56.053135 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:56.054823 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:56.055686 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:56.057719 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:56.058143 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:56.058281 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:56.062532 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:56.062538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:56.062542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:56.062578 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:56.065329 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:56.066200 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:56.066209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:56.066214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:56.066375 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:56.066806 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:56.066812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:56.066815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:56.067212 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:56.067363 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:56.068074 (+ 711us) server_negotiation.cc:299] Negotiation successful
0620 15:04:56.068339 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":233,"threads_started":1}
I20260620 15:04:56.069633 20738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:56.050078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42099 (local address 127.2.70.1:33785)
0620 15:04:56.050831 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:56.050871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:56.052608 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:56.053389 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:56.053398 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:56.053980 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:56.054610 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:56.054640 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:56.055853 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:56.055858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:56.057535 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:56.057552 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:56.058391 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:56.058398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:56.058625 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:56.059362 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:56.059428 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:56.062254 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:56.065514 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:56.065526 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:56.065548 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:56.066055 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:56.066494 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:56.066497 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:56.066499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:56.066683 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:56.067337 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:56.067343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:56.067673 (+ 330us) client_negotiation.cc:769] Sending connection context
0620 15:04:56.067913 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 15:04:56.068355 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":335,"threads_started":1}
I20260620 15:04:56.071208 20735 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42099
I20260620 15:04:56.071581 20735 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:56.072311 20735 heartbeater.cc:511] Master 127.2.70.62:42099 requested a full tablet report, sending...
I20260620 15:04:56.074671 20546 ts_manager.cc:194] Registered new tserver with Master: 513e77ddaf3e4252b7b8b124e7e91c08 (127.2.70.1:40111)
I20260620 15:04:56.076275 20546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:33785
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:04:56.112448 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42099
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:56.442950 20741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:56.443262 20741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:56.443434 20741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:56.450793 20741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:56.450937 20741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:56.451120 20741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:04:56.460961 20741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:56.462430 20741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:56.463758 20741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:56.476768 20746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:56.476852 20749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:56.476809 20747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:56.477582 20741 server_base.cc:1034] running on GCE node
I20260620 15:04:56.478197 20741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:56.479359 20741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:56.480644 20741 hybrid_clock.cc:648] HybridClock initialized: now 1781967896480584 us; error 38 us; skew 500 ppm
I20260620 15:04:56.481022 20741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:56 dist-test-slave-n5vv krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967896, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:56.489130 20741 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:04:56.490954 20741 webserver.cc:460] Webserver started at http://127.2.70.2:38843/ using document root <none> and password file <none>
I20260620 15:04:56.491780 20741 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:56.491961 20741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:56.492293 20741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:56.495384 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "e1dcb2a92d4148759b2f4fe704086ebe"
format_stamp: "Formatted at 2026-06-20 15:04:56 on dist-test-slave-n5vv"
server_key: "ec9e9a95b777b4d120e0b70a93453693"
server_key_iv: "d63468f32ae300c3e7f3ba18478d1a06"
server_key_version: "encryptionkey@0"
I20260620 15:04:56.496244 20741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "e1dcb2a92d4148759b2f4fe704086ebe"
format_stamp: "Formatted at 2026-06-20 15:04:56 on dist-test-slave-n5vv"
server_key: "ec9e9a95b777b4d120e0b70a93453693"
server_key_iv: "d63468f32ae300c3e7f3ba18478d1a06"
server_key_version: "encryptionkey@0"
I20260620 15:04:56.501761 20741 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:04:56.505421 20756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:56.507246 20741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:56.507428 20741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "e1dcb2a92d4148759b2f4fe704086ebe"
format_stamp: "Formatted at 2026-06-20 15:04:56 on dist-test-slave-n5vv"
server_key: "ec9e9a95b777b4d120e0b70a93453693"
server_key_iv: "d63468f32ae300c3e7f3ba18478d1a06"
server_key_version: "encryptionkey@0"
I20260620 15:04:56.507675 20741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:56.523344 20741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:56.527406 20741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:56.527658 20741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:56.528541 20741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:56.529909 20741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:56.529983 20741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:56.530067 20741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:56.530110 20741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:56.564466 20741 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:40931
I20260620 15:04:56.564513 20868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:40931 every 8 connection(s)
I20260620 15:04:56.565799 20741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:04:56.567329 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 20741
I20260620 15:04:56.567550 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:04:56.568076 2328 external_mini_cluster.cc:1428] Setting key c6b4b0bf9d5d9efb0aca9d20b96f1cb9
Jun 20 15:04:56 dist-test-slave-n5vv krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967896, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:56.590057 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:56.568605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:42839 (local address 127.2.70.62:42099)
0620 15:04:56.568836 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:56.568843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:56.572315 (+ 3472us) server_negotiation.cc:407] Connection header received
0620 15:04:56.574708 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:56.574718 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:56.574830 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:56.574963 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:56.577337 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:56.578355 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:56.579925 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:56.580365 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:56.580505 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:56.584141 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:56.584146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:56.584149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:56.584183 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:56.586932 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:56.587653 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:56.587658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:56.587669 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:56.587792 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:56.588291 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:56.588296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:56.588305 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:56.588579 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:56.588733 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:56.589426 (+ 693us) server_negotiation.cc:299] Negotiation successful
0620 15:04:56.589693 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260620 15:04:56.591126 20871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:56.569214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42099 (local address 127.2.70.2:42839)
0620 15:04:56.570072 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:56.570119 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:56.574393 (+ 4274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:56.575435 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:56.575449 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:56.576260 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:56.577108 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:56.577136 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:56.578519 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:56.578524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:56.579767 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:56.579788 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:56.580595 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:56.580602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:56.580850 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:56.581614 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:56.581646 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:56.583909 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:56.587071 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:56.587080 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:56.587096 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:56.587515 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:56.587904 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:56.587910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:56.587913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:56.588180 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:56.588718 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:56.588725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:56.589071 (+ 346us) client_negotiation.cc:769] Sending connection context
0620 15:04:56.589346 (+ 275us) client_negotiation.cc:240] Negotiation successful
0620 15:04:56.589913 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":393,"threads_started":1}
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
I20260620 15:04:56.592769 20869 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42099
I20260620 15:04:56.593143 20869 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:56.594017 20869 heartbeater.cc:511] Master 127.2.70.62:42099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
I20260620 15:04:56.595712 20546 ts_manager.cc:194] Registered new tserver with Master: e1dcb2a92d4148759b2f4fe704086ebe (127.2.70.2:40931)
I20260620 15:04:56.596629 20546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:42839
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:04:56.624143 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42099
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:04:56.949996 20874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:56.950327 20874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:56.950479 20874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:56.959236 20874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:56.959395 20874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:56.959583 20874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:04:56.969694 20874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:56.971036 20874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:56.972380 20874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:56.985924 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:56.985874 20882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:56.985958 20880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:56.986917 20874 server_base.cc:1034] running on GCE node
I20260620 15:04:56.987573 20874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:56.988562 20874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:56.989740 20874 hybrid_clock.cc:648] HybridClock initialized: now 1781967896989662 us; error 72 us; skew 500 ppm
I20260620 15:04:56.990113 20874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:04:56 dist-test-slave-n5vv krb5kdc[20501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967896, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:04:56.993424 20874 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:04:56.994921 20874 webserver.cc:460] Webserver started at http://127.2.70.3:42193/ using document root <none> and password file <none>
I20260620 15:04:56.995702 20874 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:56.995831 20874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:56.996212 20874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:56.999406 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "2d87648787634093a1e17e1d75112660"
format_stamp: "Formatted at 2026-06-20 15:04:56 on dist-test-slave-n5vv"
server_key: "61da15c2daee9c20f01cc856ce5ea8f3"
server_key_iv: "d77c90906a4c9ea647a187d641e3931c"
server_key_version: "encryptionkey@0"
I20260620 15:04:57.000193 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "2d87648787634093a1e17e1d75112660"
format_stamp: "Formatted at 2026-06-20 15:04:56 on dist-test-slave-n5vv"
server_key: "61da15c2daee9c20f01cc856ce5ea8f3"
server_key_iv: "d77c90906a4c9ea647a187d641e3931c"
server_key_version: "encryptionkey@0"
I20260620 15:04:57.005772 20874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 15:04:57.009608 20889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:57.011158 20874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:04:57.011369 20874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "2d87648787634093a1e17e1d75112660"
format_stamp: "Formatted at 2026-06-20 15:04:56 on dist-test-slave-n5vv"
server_key: "61da15c2daee9c20f01cc856ce5ea8f3"
server_key_iv: "d77c90906a4c9ea647a187d641e3931c"
server_key_version: "encryptionkey@0"
I20260620 15:04:57.011597 20874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:57.026731 20874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:57.032960 20874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:57.033368 20874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:57.034843 20874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:57.036788 20874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:57.036885 20874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:57.037022 20874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:57.037087 20874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:57.071025 20874 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:39585
I20260620 15:04:57.071074 21001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:39585 every 8 connection(s)
I20260620 15:04:57.072319 20874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:04:57.078713 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 20874
I20260620 15:04:57.078925 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:04:57.079361 2328 external_mini_cluster.cc:1428] Setting key 4bf03fe8f0c4b60ada36e27ce47482d9
I20260620 15:04:57.080188 20735 heartbeater.cc:503] Master 127.2.70.62:42099 was elected leader, sending a full tablet report...
Jun 20 15:04:57 dist-test-slave-n5vv krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967896, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:04:57.092424 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.075052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:52865 (local address 127.2.70.62:42099)
0620 15:04:57.075273 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.075280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.076396 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 15:04:57.078307 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.078313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.078416 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.078528 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:57.080264 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.081136 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.083016 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.083419 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.083549 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.087005 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:57.087010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:57.087013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:57.087045 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:57.089467 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.090246 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.090250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.090252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.090387 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.090800 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.090804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.090806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.091129 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:57.091269 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.091983 (+ 714us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.092161 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:04:57.093358 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.075523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:42099 (local address 127.2.70.3:52865)
0620 15:04:57.076225 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.076266 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:57.078035 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:57.078757 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:57.078765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:57.079347 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:57.080091 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.080112 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.081305 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.081317 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.082874 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.082892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.083641 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.083648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.083936 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.084613 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:04:57.084636 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:57.086811 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:57.089647 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:57.089653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:57.089665 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:57.090137 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:57.090495 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:04:57.090498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:04:57.090500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:04:57.090673 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:04:57.091278 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:57.091284 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:57.091632 (+ 348us) client_negotiation.cc:769] Sending connection context
0620 15:04:57.091853 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 15:04:57.092236 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":296,"threads_started":1}
I20260620 15:04:57.095067 21002 heartbeater.cc:347] Connected to a master server at 127.2.70.62:42099
I20260620 15:04:57.095460 21002 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:57.096342 21002 heartbeater.cc:511] Master 127.2.70.62:42099 requested a full tablet report, sending...
I20260620 15:04:57.097895 20546 ts_manager.cc:194] Registered new tserver with Master: 2d87648787634093a1e17e1d75112660 (127.2.70.3:39585)
I20260620 15:04:57.098780 20546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:52865
I20260620 15:04:57.107896 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:04:57.127051 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.112572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59548 (local address 127.2.70.62:42099)
0620 15:04:57.112770 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.112775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.113202 (+ 427us) server_negotiation.cc:407] Connection header received
0620 15:04:57.113389 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.113393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.113466 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.113593 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:57.115141 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.115949 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.117571 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.117975 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.118076 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.120957 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:57.120963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:57.120967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:57.121005 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:57.123944 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.124848 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.124852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.124855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.124976 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.125606 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.125610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.125613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.125981 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:57.126187 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.126653 (+ 466us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.126815 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:04:57.131445 20546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:04:57.139016 20546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:04:57.165827 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.153856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.2.70.3:39585)
0620 15:04:57.156876 (+ 3020us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.156883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.156910 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:04:57.156981 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.156986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.157235 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.157421 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:57.158138 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.159656 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.162880 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.164131 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.164276 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.164530 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.165259 (+ 729us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.165447 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2870,"thread_start_us":2710,"threads_started":1}
I20260620 15:04:57.166095 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.152301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.2.70.1:40111)
0620 15:04:57.153443 (+ 1142us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.153453 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.153510 (+ 57us) server_negotiation.cc:407] Connection header received
0620 15:04:57.153633 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.153642 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.153938 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.156429 (+ 2491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:57.156711 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.158625 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.162766 (+ 4141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.164319 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.164623 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.164899 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.165576 (+ 677us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.165766 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":1280,"threads_started":1}
I20260620 15:04:57.165952 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.151766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60762)
0620 15:04:57.152590 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.152644 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:57.152867 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:57.154273 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:57.154283 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:57.154318 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:57.154823 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.154839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.158829 (+ 3990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.158839 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.162564 (+ 3725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.162585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.164529 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.164537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.165122 (+ 585us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.165201 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 15:04:57.165278 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 15:04:57.165394 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":552,"spinlock_wait_cycles":14080,"thread_start_us":675,"threads_started":1}
I20260620 15:04:57.166795 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.152988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.2.70.2:40931)
0620 15:04:57.153552 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.153561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.154271 (+ 710us) server_negotiation.cc:407] Connection header received
0620 15:04:57.154587 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.154595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.154888 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.155062 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:57.156776 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.158630 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.162298 (+ 3668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.164223 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.164382 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.164603 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.166287 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.166469 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":261,"threads_started":1}
I20260620 15:04:57.166987 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.153283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:39585 (local address 127.0.0.1:34782)
0620 15:04:57.156462 (+ 3179us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.156481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:57.156592 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:57.157572 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:57.157578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:57.157603 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:57.157940 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.157949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.159917 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.159924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.162722 (+ 2798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.162735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.164695 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.164702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.164933 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.164999 (+ 66us) client_negotiation.cc:769] Sending connection context
0620 15:04:57.165135 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 15:04:57.165360 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3042,"mutex_wait_us":515,"thread_start_us":756,"threads_started":1}
I20260620 15:04:57.167045 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.152479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:40931 (local address 127.0.0.1:33210)
0620 15:04:57.154166 (+ 1687us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.154185 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:57.154371 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:57.155353 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:57.155357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:57.155387 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:57.155900 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.155915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.158903 (+ 2988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.158909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.162062 (+ 3153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.162077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.164763 (+ 2686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.164769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.165053 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.165192 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 15:04:57.165290 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 15:04:57.165440 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"mutex_wait_us":817,"thread_start_us":3705,"threads_started":1}
I20260620 15:04:57.170686 20937 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd55b8a43d14e2fa6b66e64df674c55 (DEFAULT_TABLE table=test-table [id=b229a58e4b3e4b7b802c0712c2c00408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:57.170605 20804 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd55b8a43d14e2fa6b66e64df674c55 (DEFAULT_TABLE table=test-table [id=b229a58e4b3e4b7b802c0712c2c00408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:57.170605 20670 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd55b8a43d14e2fa6b66e64df674c55 (DEFAULT_TABLE table=test-table [id=b229a58e4b3e4b7b802c0712c2c00408]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:04:57.171964 20937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd55b8a43d14e2fa6b66e64df674c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:57.172031 20804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd55b8a43d14e2fa6b66e64df674c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:57.172021 20670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd55b8a43d14e2fa6b66e64df674c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:57.184784 21020 tablet_bootstrap.cc:492] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Bootstrap starting.
I20260620 15:04:57.188130 21020 tablet_bootstrap.cc:654] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:57.189067 21022 tablet_bootstrap.cc:492] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08: Bootstrap starting.
I20260620 15:04:57.189980 21021 tablet_bootstrap.cc:492] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Bootstrap starting.
I20260620 15:04:57.190006 21020 log.cc:824] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:57.193455 21020 tablet_bootstrap.cc:492] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: No bootstrap required, opened a new log
I20260620 15:04:57.193820 21020 ts_tablet_manager.cc:1397] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260620 15:04:57.194444 21022 tablet_bootstrap.cc:654] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:57.194571 21021 tablet_bootstrap.cc:654] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:57.195838 21021 log.cc:824] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:57.195838 21022 log.cc:824] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:57.199478 21022 tablet_bootstrap.cc:492] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08: No bootstrap required, opened a new log
I20260620 15:04:57.199473 21021 tablet_bootstrap.cc:492] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: No bootstrap required, opened a new log
I20260620 15:04:57.199791 21021 ts_tablet_manager.cc:1397] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260620 15:04:57.199788 21022 ts_tablet_manager.cc:1397] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260620 15:04:57.202203 21020 raft_consensus.cc:348] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.202618 21020 raft_consensus.cc:374] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:57.202720 21020 raft_consensus.cc:729] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d87648787634093a1e17e1d75112660, State: Initialized, Role: FOLLOWER
I20260620 15:04:57.203389 21020 consensus_queue.cc:260] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.205565 21002 heartbeater.cc:503] Master 127.2.70.62:42099 was elected leader, sending a full tablet report...
I20260620 15:04:57.206355 21020 ts_tablet_manager.cc:1428] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Time spent starting tablet: real 0.012s user 0.001s sys 0.010s
I20260620 15:04:57.211202 21022 raft_consensus.cc:348] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.211202 21021 raft_consensus.cc:348] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.211720 21022 raft_consensus.cc:374] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:57.211719 21021 raft_consensus.cc:374] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:57.211835 21022 raft_consensus.cc:729] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513e77ddaf3e4252b7b8b124e7e91c08, State: Initialized, Role: FOLLOWER
I20260620 15:04:57.211834 21021 raft_consensus.cc:729] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1dcb2a92d4148759b2f4fe704086ebe, State: Initialized, Role: FOLLOWER
I20260620 15:04:57.212504 21022 consensus_queue.cc:260] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.212496 21021 consensus_queue.cc:260] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.214272 20869 heartbeater.cc:503] Master 127.2.70.62:42099 was elected leader, sending a full tablet report...
I20260620 15:04:57.214541 21021 ts_tablet_manager.cc:1428] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260620 15:04:57.214763 21022 ts_tablet_manager.cc:1428] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260620 15:04:57.299388 20736 tablet.cc:2369] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:04:57.303079 21028 raft_consensus.cc:482] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:04:57.303428 21028 raft_consensus.cc:504] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.305546 21028 leader_election.cc:288] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 513e77ddaf3e4252b7b8b124e7e91c08 (127.2.70.1:40111), 2d87648787634093a1e17e1d75112660 (127.2.70.3:39585)
I20260620 15:04:57.313189 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.306123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:33033 (local address 127.2.70.1:40111)
0620 15:04:57.306409 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.306417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.306853 (+ 436us) server_negotiation.cc:407] Connection header received
0620 15:04:57.307133 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.307140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.307233 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.307336 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:57.308086 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.309291 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.311385 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.312188 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.312344 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.312457 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.312807 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.312934 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 15:04:57.313328 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.307120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:50689 (local address 127.2.70.3:39585)
0620 15:04:57.307352 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.307358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.307670 (+ 312us) server_negotiation.cc:407] Connection header received
0620 15:04:57.307903 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.307909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.307996 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.308082 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:04:57.308664 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.309721 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.311541 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.312327 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.312476 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.312590 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.312928 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.313066 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:04:57.313467 21030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.306753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:39585 (local address 127.2.70.2:50689)
0620 15:04:57.307565 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.307580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:57.307741 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:57.308104 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:57.308108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:57.308133 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:57.308494 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.308504 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.309856 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.309862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.311386 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.311398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.312550 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.312557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.312764 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.312787 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:04:57.312891 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:04:57.313016 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"spinlock_wait_cycles":29952,"thread_start_us":503,"threads_started":1}
I20260620 15:04:57.313680 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.305923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.2.70.2:33033)
0620 15:04:57.306730 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.306757 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:57.306973 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:57.307447 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:57.307452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:57.307482 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:04:57.307918 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.307931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.309467 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.309473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.311218 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:57.311237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.312406 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.312412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:57.312603 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.312650 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 15:04:57.312783 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 15:04:57.312920 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":426,"threads_started":1}
I20260620 15:04:57.314599 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd55b8a43d14e2fa6b66e64df674c55" candidate_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d87648787634093a1e17e1d75112660" is_pre_election: true
I20260620 15:04:57.314669 20690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd55b8a43d14e2fa6b66e64df674c55" candidate_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" is_pre_election: true
I20260620 15:04:57.315156 20957 raft_consensus.cc:2454] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dcb2a92d4148759b2f4fe704086ebe in term 0.
I20260620 15:04:57.315133 20690 raft_consensus.cc:2454] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1dcb2a92d4148759b2f4fe704086ebe in term 0.
I20260620 15:04:57.315907 20758 leader_election.cc:302] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513e77ddaf3e4252b7b8b124e7e91c08, e1dcb2a92d4148759b2f4fe704086ebe; no voters:
I20260620 15:04:57.316419 21028 raft_consensus.cc:2781] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:04:57.316599 21028 raft_consensus.cc:482] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:04:57.316707 21028 raft_consensus.cc:3037] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 0 FOLLOWER]: Advancing to term 1
W20260620 15:04:57.318049 20870 tablet.cc:2369] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:04:57.319227 21028 raft_consensus.cc:504] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.319885 21028 leader_election.cc:288] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [CANDIDATE]: Term 1 election: Requested vote from peers 513e77ddaf3e4252b7b8b124e7e91c08 (127.2.70.1:40111), 2d87648787634093a1e17e1d75112660 (127.2.70.3:39585)
I20260620 15:04:57.320436 20690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd55b8a43d14e2fa6b66e64df674c55" candidate_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "513e77ddaf3e4252b7b8b124e7e91c08"
I20260620 15:04:57.320670 20690 raft_consensus.cc:3037] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:57.320601 20957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd55b8a43d14e2fa6b66e64df674c55" candidate_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d87648787634093a1e17e1d75112660"
I20260620 15:04:57.320906 20957 raft_consensus.cc:3037] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:57.323051 20690 raft_consensus.cc:2454] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dcb2a92d4148759b2f4fe704086ebe in term 1.
I20260620 15:04:57.323228 20957 raft_consensus.cc:2454] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1dcb2a92d4148759b2f4fe704086ebe in term 1.
I20260620 15:04:57.323559 20758 leader_election.cc:302] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513e77ddaf3e4252b7b8b124e7e91c08, e1dcb2a92d4148759b2f4fe704086ebe; no voters:
I20260620 15:04:57.323912 21028 raft_consensus.cc:2781] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 1 FOLLOWER]: Leader election won for term 1
W20260620 15:04:57.324011 21003 tablet.cc:2369] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:04:57.324465 21028 raft_consensus.cc:686] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 1 LEADER]: Becoming Leader. State: Replica: e1dcb2a92d4148759b2f4fe704086ebe, State: Running, Role: LEADER
I20260620 15:04:57.324967 21028 consensus_queue.cc:237] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } }
I20260620 15:04:57.330004 20545 catalog_manager.cc:5495] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe reported cstate change: term changed from 0 to 1, leader changed from <none> to e1dcb2a92d4148759b2f4fe704086ebe (127.2.70.2). New cstate: current_term: 1 leader_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1dcb2a92d4148759b2f4fe704086ebe" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 40931 } health_report { overall_health: HEALTHY } } }
I20260620 15:04:57.359387 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.353651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.2.70.2:40931)
0620 15:04:57.353976 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.353983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.354003 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:04:57.354215 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.354220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.354340 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.354503 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:04:57.354919 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.355642 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.357087 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.357566 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.357701 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.358171 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:04:57.358694 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:04:57.358830 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.358959 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.359024 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260620 15:04:57.368112 20690 raft_consensus.cc:1261] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 1 FOLLOWER]: Refusing update from remote peer e1dcb2a92d4148759b2f4fe704086ebe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:04:57.368112 20957 raft_consensus.cc:1261] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 1 FOLLOWER]: Refusing update from remote peer e1dcb2a92d4148759b2f4fe704086ebe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:04:57.369114 21031 consensus_queue.cc:1035] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "513e77ddaf3e4252b7b8b124e7e91c08" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 40111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:04:57.369526 21028 consensus_queue.cc:1035] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d87648787634093a1e17e1d75112660" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 39585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:04:57.382079 21034 mvcc.cc:205] Tried to move back new op lower bound from 7298940507608526848 to 7298940507447349248. Current Snapshot: MvccSnapshot[applied={T|T < 7298940507608526848}]
I20260620 15:04:57.396966 20546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:04:57.397311 20546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:04:57.400832 20546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b272517ba3aa467fa197b1b66ea7b731: Sending DeleteTablet for 3 replicas of tablet 6fd55b8a43d14e2fa6b66e64df674c55
I20260620 15:04:57.402004 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd55b8a43d14e2fa6b66e64df674c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:57 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:60762
I20260620 15:04:57.402133 20937 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd55b8a43d14e2fa6b66e64df674c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:57 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:34782
I20260620 15:04:57.403105 20804 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd55b8a43d14e2fa6b66e64df674c55 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:04:57 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:33210
I20260620 15:04:57.403532 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 20607
I20260620 15:04:57.428328 21046 tablet_replica.cc:330] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: stopping tablet replica
Jun 20 15:04:57 dist-test-slave-n5vv krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:04:57.441092 21046 raft_consensus.cc:2229] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:04:57.442081 21046 raft_consensus.cc:2258] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:57.442744 21044 tablet_replica.cc:330] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08: stopping tablet replica
I20260620 15:04:57.443303 21044 raft_consensus.cc:2229] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:04:57.454070 21044 raft_consensus.cc:2258] T 6fd55b8a43d14e2fa6b66e64df674c55 P 513e77ddaf3e4252b7b8b124e7e91c08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:57.452023 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.405039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.2.70.1:40111)
0620 15:04:57.405273 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.405282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.426704 (+ 21422us) server_negotiation.cc:407] Connection header received
0620 15:04:57.427031 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.427041 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.427204 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.427343 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:57.429866 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.431428 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.434183 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.435088 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.435216 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.442293 (+ 7077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:57.442319 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:57.442353 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:57.442427 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:57.447435 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.448577 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.448592 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.448598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.448811 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.449374 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.449381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.449387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.449994 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:57.450351 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.451100 (+ 749us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.451418 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:04:57.455042 20730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:57.458024 21045 tablet_replica.cc:330] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: stopping tablet replica
I20260620 15:04:57.458513 21045 raft_consensus.cc:2229] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:04:57.459061 21045 raft_consensus.cc:2258] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:04:57.461426 21045 ts_tablet_manager.cc:1905] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:57.463250 21046 ts_tablet_manager.cc:1905] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:04:57.468467 21045 ts_tablet_manager.cc:1918] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:57.468634 21045 log.cc:1196] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/6fd55b8a43d14e2fa6b66e64df674c55
I20260620 15:04:57.469161 21045 ts_tablet_manager.cc:1939] T 6fd55b8a43d14e2fa6b66e64df674c55 P 2d87648787634093a1e17e1d75112660: Deleting consensus metadata
I20260620 15:04:57.472252 20531 catalog_manager.cc:4841] TS 2d87648787634093a1e17e1d75112660 (127.2.70.3:39585): tablet 6fd55b8a43d14e2fa6b66e64df674c55 (table test-table [id=b229a58e4b3e4b7b802c0712c2c00408]) successfully deleted
I20260620 15:04:57.475189 21046 ts_tablet_manager.cc:1918] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:04:57.475486 21046 log.cc:1196] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/6fd55b8a43d14e2fa6b66e64df674c55
I20260620 15:04:57.476044 21046 ts_tablet_manager.cc:1939] T 6fd55b8a43d14e2fa6b66e64df674c55 P e1dcb2a92d4148759b2f4fe704086ebe: Deleting consensus metadata
I20260620 15:04:57.477579 20530 catalog_manager.cc:4841] TS e1dcb2a92d4148759b2f4fe704086ebe (127.2.70.2:40931): tablet 6fd55b8a43d14e2fa6b66e64df674c55 (table test-table [id=b229a58e4b3e4b7b802c0712c2c00408]) successfully deleted
I20260620 15:04:57.586047 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 20607
I20260620 15:04:57.604199 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 20741
W20260620 15:04:57.604199 20531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.1:40111 (error 108)
I20260620 15:04:57.606616 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.606024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60786)
0620 15:04:57.606310 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.606422 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260620 15:04:57.606920 20531 catalog_manager.cc:4584] TS 513e77ddaf3e4252b7b8b124e7e91c08 (127.2.70.1:40111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fd55b8a43d14e2fa6b66e64df674c55: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Jun 20 15:04:57 dist-test-slave-n5vv krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:04:57.622257 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.605459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33232 (local address 127.2.70.2:40931)
0620 15:04:57.605687 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.605694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.605715 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:04:57.605869 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.605877 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.605961 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.606071 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:57.607692 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.608872 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.610186 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.610907 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.611022 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.615251 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:57.615268 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:57.615289 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:57.615336 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:57.619315 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.619997 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.620008 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.620015 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.620181 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.620547 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.620553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.620557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.621026 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:57.621401 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.621599 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.621843 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260620 15:04:57.623065 20864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:57.660670 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.660105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60790)
0620 15:04:57.660388 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.660500 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 15:04:57.661772 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.661414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60802)
0620 15:04:57.661590 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.661671 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 15:04:57.722666 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.722111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60808)
0620 15:04:57.722406 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.722506 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260620 15:04:57.723763 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.723427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60824)
0620 15:04:57.723574 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.723645 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 15:04:57.741547 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 20741
I20260620 15:04:57.760030 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 20874
Jun 20 15:04:57 dist-test-slave-n5vv krb5kdc[20501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:04:57.775867 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.760813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34798 (local address 127.2.70.3:39585)
0620 15:04:57.761008 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:57.761016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:57.761063 (+ 47us) server_negotiation.cc:407] Connection header received
0620 15:04:57.761362 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:57.761366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:57.761431 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:57.761561 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:57.762849 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.763582 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.765096 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:57.765648 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:57.765752 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:57.770138 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:57.770152 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:57.770170 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:04:57.770207 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:57.772894 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.773679 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.773686 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.773688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.773837 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:04:57.774208 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:04:57.774211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:04:57.774213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:04:57.774624 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:57.774972 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:57.775265 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 15:04:57.775487 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:04:57.776563 20997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:57.804519 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.804015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60832)
0620 15:04:57.804235 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.804339 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 15:04:57.805665 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.805211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60842)
0620 15:04:57.805422 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.805541 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260620 15:04:57.865073 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 20874
I20260620 15:04:57.881639 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 20514
I20260620 15:04:57.882851 20576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:04:57.994458 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 20514
I20260620 15:04:57.994482 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:57.993723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:40111 (local address 127.0.0.1:60858)
0620 15:04:57.993948 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:57.994047 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:40111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
2026-06-20T15:04:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4362 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T15:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:04:58Z Disabled control of system clock
I20260620 15:04:58.071607 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46697
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:46697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:04:58.407815 21056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:58.408098 21056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:58.408248 21056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:58.415637 21056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:04:58.415750 21056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:58.415845 21056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:58.415939 21056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:04:58.416000 21056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:04:58.425427 21056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:46697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:46697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:58.426748 21056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:58.428544 21056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:58.440258 21061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:58.440173 21062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:58.440215 21064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:58.441606 21056 server_base.cc:1034] running on GCE node
I20260620 15:04:58.442317 21056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:58.443758 21056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:58.445027 21056 hybrid_clock.cc:648] HybridClock initialized: now 1781967898444974 us; error 37 us; skew 500 ppm
I20260620 15:04:58.445405 21056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:04:58.447604 21056 webserver.cc:460] Webserver started at http://127.2.70.62:45603/ using document root <none> and password file <none>
I20260620 15:04:58.448297 21056 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:58.448428 21056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:58.448765 21056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:58.451758 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "00633dbaaaa948b08d4a4122bac328b6"
format_stamp: "Formatted at 2026-06-20 15:04:58 on dist-test-slave-n5vv"
server_key: "719ba02e27160465511fa83bb9c0732d"
server_key_iv: "b5ea9ad3545503ec4d86d97cc3d1a564"
server_key_version: "encryptionkey@0"
I20260620 15:04:58.452605 21056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "00633dbaaaa948b08d4a4122bac328b6"
format_stamp: "Formatted at 2026-06-20 15:04:58 on dist-test-slave-n5vv"
server_key: "719ba02e27160465511fa83bb9c0732d"
server_key_iv: "b5ea9ad3545503ec4d86d97cc3d1a564"
server_key_version: "encryptionkey@0"
I20260620 15:04:58.457789 21056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:04:58.461169 21070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:58.462569 21056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:58.462759 21056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "00633dbaaaa948b08d4a4122bac328b6"
format_stamp: "Formatted at 2026-06-20 15:04:58 on dist-test-slave-n5vv"
server_key: "719ba02e27160465511fa83bb9c0732d"
server_key_iv: "b5ea9ad3545503ec4d86d97cc3d1a564"
server_key_version: "encryptionkey@0"
I20260620 15:04:58.462988 21056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:58.485741 21056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:58.486644 21056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:58.486944 21056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:58.505616 21056 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:46697
I20260620 15:04:58.505637 21121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:46697 every 8 connection(s)
I20260620 15:04:58.507302 21056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:04:58.512046 21122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:04:58.514601 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 21056
I20260620 15:04:58.514772 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:04:58.515326 2328 external_mini_cluster.cc:1428] Setting key 5bb18a040d3c2e4f7b35821193ea5907
I20260620 15:04:58.526032 21122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6: Bootstrap starting.
I20260620 15:04:58.528549 21125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:58.517284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.2.70.62:46697)
0620 15:04:58.518265 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:58.518276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:58.518319 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:04:58.519968 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:58.519990 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:58.520476 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:58.520863 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:58.521439 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:58.522750 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:58.524195 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:58.524872 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:58.525073 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:58.526295 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:58.526302 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:58.526315 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:58.526870 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:58.526948 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:58.526969 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:58.527107 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:58.527299 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 15:04:58.527620 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":355,"threads_started":1}
I20260620 15:04:58.529451 21122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6: Neither blocks nor log segments found. Creating new log.
I20260620 15:04:58.530579 21122 log.cc:824] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6: Log is configured to *not* fsync() on all Append() calls
I20260620 15:04:58.533262 21122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6: No bootstrap required, opened a new log
I20260620 15:04:58.540354 21122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00633dbaaaa948b08d4a4122bac328b6" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46697 } }
I20260620 15:04:58.540890 21122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:04:58.541002 21122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00633dbaaaa948b08d4a4122bac328b6, State: Initialized, Role: FOLLOWER
I20260620 15:04:58.541705 21122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00633dbaaaa948b08d4a4122bac328b6" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46697 } }
I20260620 15:04:58.541914 21122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:04:58.542012 21122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:04:58.542130 21122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:04:58.544173 21122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00633dbaaaa948b08d4a4122bac328b6" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46697 } }
I20260620 15:04:58.544629 21122 leader_election.cc:302] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00633dbaaaa948b08d4a4122bac328b6; no voters:
I20260620 15:04:58.545121 21122 leader_election.cc:288] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:04:58.545356 21127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:04:58.546849 21127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [term 1 LEADER]: Becoming Leader. State: Replica: 00633dbaaaa948b08d4a4122bac328b6, State: Running, Role: LEADER
I20260620 15:04:58.547370 21127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00633dbaaaa948b08d4a4122bac328b6" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46697 } }
I20260620 15:04:58.547572 21122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:04:58.550505 21128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00633dbaaaa948b08d4a4122bac328b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00633dbaaaa948b08d4a4122bac328b6" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46697 } } }
I20260620 15:04:58.551247 21128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:58.550513 21129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00633dbaaaa948b08d4a4122bac328b6. Latest consensus state: current_term: 1 leader_uuid: "00633dbaaaa948b08d4a4122bac328b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00633dbaaaa948b08d4a4122bac328b6" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 46697 } } }
I20260620 15:04:58.551860 21129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6 [sys.catalog]: This master's current role is: LEADER
I20260620 15:04:58.554858 21137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:04:58.559437 21137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:04:58.567790 21137 catalog_manager.cc:1347] Generated new cluster ID: 2dff28447ec14fe799ef05e751c2c372
I20260620 15:04:58.567903 21137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:04:58.586593 21137 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:04:58.587821 21137 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:04:58.598209 21137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6: Generated new TSK 0
I20260620 15:04:58.599274 21137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:04:58.615747 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46697
--builtin_ntp_servers=127.2.70.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:04:58.950033 21146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:58.950310 21146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:58.950459 21146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:58.958053 21146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:58.958154 21146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:58.958350 21146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:04:58.969173 21146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:58.970507 21146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:58.972275 21146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:58.985415 21151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:58.985302 21152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:58.985466 21154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:58.986274 21146 server_base.cc:1034] running on GCE node
I20260620 15:04:58.986883 21146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:58.987816 21146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:58.989019 21146 hybrid_clock.cc:648] HybridClock initialized: now 1781967898988979 us; error 39 us; skew 500 ppm
I20260620 15:04:58.989331 21146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:04:58.991576 21146 webserver.cc:460] Webserver started at http://127.2.70.1:35933/ using document root <none> and password file <none>
I20260620 15:04:58.992240 21146 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:58.992359 21146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:58.992688 21146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:58.995510 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "5a998701b12a4ce6872a098ef7982b13"
format_stamp: "Formatted at 2026-06-20 15:04:58 on dist-test-slave-n5vv"
server_key: "a2640d57f31fadb05b2a9825b8f068e0"
server_key_iv: "7872bb14a886c925b47c7b31bee94f82"
server_key_version: "encryptionkey@0"
I20260620 15:04:58.996280 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "5a998701b12a4ce6872a098ef7982b13"
format_stamp: "Formatted at 2026-06-20 15:04:58 on dist-test-slave-n5vv"
server_key: "a2640d57f31fadb05b2a9825b8f068e0"
server_key_iv: "7872bb14a886c925b47c7b31bee94f82"
server_key_version: "encryptionkey@0"
I20260620 15:04:59.001287 21146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:04:59.004896 21160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.006573 21146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:04:59.006734 21146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "5a998701b12a4ce6872a098ef7982b13"
format_stamp: "Formatted at 2026-06-20 15:04:58 on dist-test-slave-n5vv"
server_key: "a2640d57f31fadb05b2a9825b8f068e0"
server_key_iv: "7872bb14a886c925b47c7b31bee94f82"
server_key_version: "encryptionkey@0"
I20260620 15:04:59.007016 21146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:59.018939 21146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:59.019811 21146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:59.020298 21146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:59.021317 21146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:59.022897 21146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:59.023032 21146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.023175 21146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:59.023274 21146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.057934 21146 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:33827
I20260620 15:04:59.057976 21272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:33827 every 8 connection(s)
I20260620 15:04:59.059633 21146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:04:59.063441 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 21146
I20260620 15:04:59.063606 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:04:59.064170 2328 external_mini_cluster.cc:1428] Setting key 884e277dd935879a7100b20f92da42ca
I20260620 15:04:59.067196 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46697
--builtin_ntp_servers=127.2.70.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:04:59.074589 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:59.063183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:46747 (local address 127.2.70.62:46697)
0620 15:04:59.063651 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:59.063655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:59.065557 (+ 1902us) server_negotiation.cc:407] Connection header received
0620 15:04:59.067226 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:59.067232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:59.067318 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:59.067493 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:59.068549 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.069766 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.071381 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.072149 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.072293 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:59.073587 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:59.073596 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:59.073603 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:59.073629 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:59.073683 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:59.073701 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:59.073794 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:59.074019 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 15:04:59.074133 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":252,"threads_started":1}
I20260620 15:04:59.075381 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:59.063830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46697 (local address 127.2.70.1:46747)
0620 15:04:59.064864 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:59.064920 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:59.066943 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:59.067702 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:59.067718 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:59.067815 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:59.068366 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:59.068383 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.069954 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.069959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:59.071233 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:59.071245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.072353 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.072358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:59.072552 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:59.073245 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:04:59.073248 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:59.073291 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:04:59.073369 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:59.073799 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:59.073808 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:59.073814 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:04:59.073981 (+ 167us) client_negotiation.cc:240] Negotiation successful
0620 15:04:59.074266 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":303,"threads_started":1}
I20260620 15:04:59.076988 21273 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46697
I20260620 15:04:59.077306 21273 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:59.078212 21273 heartbeater.cc:511] Master 127.2.70.62:46697 requested a full tablet report, sending...
I20260620 15:04:59.080420 21087 ts_manager.cc:194] Registered new tserver with Master: 5a998701b12a4ce6872a098ef7982b13 (127.2.70.1:33827)
I20260620 15:04:59.082010 21087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.1:46747
W20260620 15:04:59.399187 21277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:59.399530 21277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:59.399682 21277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:59.407255 21277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:59.407403 21277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:59.407580 21277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:04:59.417707 21277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:59.419063 21277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:59.420855 21277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:59.434113 21285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:59.434077 21283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:59.434151 21282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:59.435840 21277 server_base.cc:1034] running on GCE node
I20260620 15:04:59.436486 21277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:59.437541 21277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:59.438778 21277 hybrid_clock.cc:648] HybridClock initialized: now 1781967899438716 us; error 59 us; skew 500 ppm
I20260620 15:04:59.439100 21277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:04:59.441313 21277 webserver.cc:460] Webserver started at http://127.2.70.2:34893/ using document root <none> and password file <none>
I20260620 15:04:59.442094 21277 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:59.442224 21277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:59.442570 21277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:59.445364 21277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "1ee19c9cfe564330bc0da3f4b14eb519"
format_stamp: "Formatted at 2026-06-20 15:04:59 on dist-test-slave-n5vv"
server_key: "8096374951c1c5b6c3b9e3d952cb37e5"
server_key_iv: "c83e9411fca946b09ca2deeef547b7c0"
server_key_version: "encryptionkey@0"
I20260620 15:04:59.446183 21277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "1ee19c9cfe564330bc0da3f4b14eb519"
format_stamp: "Formatted at 2026-06-20 15:04:59 on dist-test-slave-n5vv"
server_key: "8096374951c1c5b6c3b9e3d952cb37e5"
server_key_iv: "c83e9411fca946b09ca2deeef547b7c0"
server_key_version: "encryptionkey@0"
I20260620 15:04:59.451154 21277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:04:59.454798 21291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.456178 21277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:04:59.456305 21277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "1ee19c9cfe564330bc0da3f4b14eb519"
format_stamp: "Formatted at 2026-06-20 15:04:59 on dist-test-slave-n5vv"
server_key: "8096374951c1c5b6c3b9e3d952cb37e5"
server_key_iv: "c83e9411fca946b09ca2deeef547b7c0"
server_key_version: "encryptionkey@0"
I20260620 15:04:59.456490 21277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:59.486129 21277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:59.486946 21277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:59.487179 21277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:59.488086 21277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:59.489581 21277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:59.489670 21277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.489756 21277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:59.489813 21277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.524607 21277 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:46809
I20260620 15:04:59.524654 21403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:46809 every 8 connection(s)
I20260620 15:04:59.525976 21277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:04:59.534554 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 21277
I20260620 15:04:59.534734 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:04:59.535171 2328 external_mini_cluster.cc:1428] Setting key aabc1d637bebef9ce993c9f378e11dcf
I20260620 15:04:59.538453 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46697
--builtin_ntp_servers=127.2.70.20:46121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:04:59.539105 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:59.528571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:45557 (local address 127.2.70.62:46697)
0620 15:04:59.528776 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:59.528782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:59.530300 (+ 1518us) server_negotiation.cc:407] Connection header received
0620 15:04:59.532085 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:59.532089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:59.532155 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:59.532247 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:59.533363 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.534204 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.535939 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.536435 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.536530 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:59.538133 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:04:59.538143 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:04:59.538151 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:04:59.538175 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:04:59.538221 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:04:59.538240 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:04:59.538345 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:04:59.538558 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 15:04:59.538670 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:04:59.539990 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:59.529220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46697 (local address 127.2.70.2:45557)
0620 15:04:59.530145 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:59.530181 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:59.531826 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:59.532465 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:59.532473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:59.532561 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:59.533206 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:59.533228 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.534329 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.534333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:59.535758 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:59.535774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.536653 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.536660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:59.536976 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:04:59.537798 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:04:59.537800 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:04:59.537843 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:04:59.537924 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:04:59.538340 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:04:59.538347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:04:59.538353 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:04:59.538505 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 15:04:59.538892 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":316,"threads_started":1}
I20260620 15:04:59.541420 21404 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46697
I20260620 15:04:59.541761 21404 heartbeater.cc:464] Registering TS with master...
I20260620 15:04:59.542718 21404 heartbeater.cc:511] Master 127.2.70.62:46697 requested a full tablet report, sending...
I20260620 15:04:59.544247 21087 ts_manager.cc:194] Registered new tserver with Master: 1ee19c9cfe564330bc0da3f4b14eb519 (127.2.70.2:46809)
I20260620 15:04:59.545037 21087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.2:45557
W20260620 15:04:59.873874 21407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:04:59.874233 21407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:04:59.874430 21407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:04:59.882599 21407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:04:59.882756 21407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:04:59.882943 21407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:04:59.892302 21407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:46121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:46697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:04:59.893702 21407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:04:59.895529 21407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:04:59.908839 21412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:59.908792 21413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:04:59.908814 21415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:04:59.910527 21407 server_base.cc:1034] running on GCE node
I20260620 15:04:59.911314 21407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:04:59.912410 21407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:04:59.913664 21407 hybrid_clock.cc:648] HybridClock initialized: now 1781967899913621 us; error 40 us; skew 500 ppm
I20260620 15:04:59.914045 21407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:04:59.916380 21407 webserver.cc:460] Webserver started at http://127.2.70.3:33553/ using document root <none> and password file <none>
I20260620 15:04:59.917096 21407 fs_manager.cc:359] Metadata directory not provided
I20260620 15:04:59.917207 21407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:04:59.917613 21407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:04:59.920460 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "020ff46557b1410bb703634412617f32"
format_stamp: "Formatted at 2026-06-20 15:04:59 on dist-test-slave-n5vv"
server_key: "69a5a2f405e58955db282a4f0779c1dc"
server_key_iv: "fd1f7393cc30e6b3c2584b8d19229573"
server_key_version: "encryptionkey@0"
I20260620 15:04:59.921276 21407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "020ff46557b1410bb703634412617f32"
format_stamp: "Formatted at 2026-06-20 15:04:59 on dist-test-slave-n5vv"
server_key: "69a5a2f405e58955db282a4f0779c1dc"
server_key_iv: "fd1f7393cc30e6b3c2584b8d19229573"
server_key_version: "encryptionkey@0"
I20260620 15:04:59.926359 21407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260620 15:04:59.929782 21421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.931051 21407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:04:59.931214 21407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "020ff46557b1410bb703634412617f32"
format_stamp: "Formatted at 2026-06-20 15:04:59 on dist-test-slave-n5vv"
server_key: "69a5a2f405e58955db282a4f0779c1dc"
server_key_iv: "fd1f7393cc30e6b3c2584b8d19229573"
server_key_version: "encryptionkey@0"
I20260620 15:04:59.931442 21407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:04:59.949692 21407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:04:59.950472 21407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:04:59.950743 21407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:04:59.951689 21407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:04:59.953284 21407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:04:59.953395 21407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.953549 21407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:04:59.953640 21407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:04:59.988016 21407 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:37651
I20260620 15:04:59.988072 21533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:37651 every 8 connection(s)
I20260620 15:04:59.989382 21407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:04:59.994234 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 21407
I20260620 15:04:59.994415 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:04:59.994963 2328 external_mini_cluster.cc:1428] Setting key 438f88de2fcfa37ff10200652d53ebf6
I20260620 15:05:00.002532 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:59.991946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:46843 (local address 127.2.70.62:46697)
0620 15:04:59.992165 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:04:59.992171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:04:59.993801 (+ 1630us) server_negotiation.cc:407] Connection header received
0620 15:04:59.996256 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:04:59.996263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:04:59.996345 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:04:59.996434 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:04:59.997657 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.998421 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.999897 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.000286 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.000388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.001715 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:00.001721 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:00.001724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:00.001741 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:00.001774 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:00.001787 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:00.001888 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.002177 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.002277 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:05:00.003459 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:04:59.992567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:46697 (local address 127.2.70.3:46843)
0620 15:04:59.993637 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0620 15:04:59.993680 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 15:04:59.995963 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:04:59.996721 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:04:59.996733 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:04:59.996851 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:04:59.997469 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:59.997494 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:04:59.998558 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:04:59.998562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:04:59.999772 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:04:59.999783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.000476 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.000481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.000671 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.001328 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:05:00.001331 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:00.001380 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:05:00.001466 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:00.001899 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:00.001906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:00.001912 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 15:05:00.002051 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:05:00.002360 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":377,"threads_started":1}
I20260620 15:05:00.005283 21534 heartbeater.cc:347] Connected to a master server at 127.2.70.62:46697
I20260620 15:05:00.005676 21534 heartbeater.cc:464] Registering TS with master...
I20260620 15:05:00.006490 21534 heartbeater.cc:511] Master 127.2.70.62:46697 requested a full tablet report, sending...
I20260620 15:05:00.008020 21087 ts_manager.cc:194] Registered new tserver with Master: 020ff46557b1410bb703634412617f32 (127.2.70.3:37651)
I20260620 15:05:00.008872 21087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.3:46843
I20260620 15:05:00.010360 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:05:00.021231 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.014999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60076 (local address 127.2.70.62:46697)
0620 15:05:00.015269 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.015278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.015579 (+ 301us) server_negotiation.cc:407] Connection header received
0620 15:05:00.015782 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.015794 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.015867 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.015985 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:00.016471 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.017187 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.018734 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.019158 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.019299 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.020462 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:00.020467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:00.020470 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:00.020489 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:00.020519 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:00.020532 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:00.020682 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.020868 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.020935 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:05:00.025415 21087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:05:00.032296 21087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:05:00.071077 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.044961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.2.70.2:46809)
0620 15:05:00.058567 (+ 13606us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.058577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.060761 (+ 2184us) server_negotiation.cc:407] Connection header received
0620 15:05:00.060912 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.060924 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.061319 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.061581 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:00.063317 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.065346 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.068639 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.069909 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.070061 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.070309 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.070580 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.070747 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13394,"thread_start_us":20531,"threads_started":1}
I20260620 15:05:00.071130 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.044619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:46809 (local address 127.0.0.1:43042)
0620 15:05:00.060476 (+ 15857us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:00.060495 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:00.060631 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:00.062518 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:00.062529 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:00.062572 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:00.063060 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.063075 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.065555 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.065563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.068430 (+ 2867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.068443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.070111 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.070117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.070334 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.070398 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 15:05:00.070537 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 15:05:00.070722 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15737,"mutex_wait_us":35,"thread_start_us":21425,"threads_started":1}
I20260620 15:05:00.072417 21547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.044525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:37651 (local address 127.0.0.1:35284)
0620 15:05:00.060073 (+ 15548us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:00.060121 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:00.060333 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:00.063256 (+ 2923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:00.063262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:00.063285 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:00.063669 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.063679 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.066141 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.066149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.069118 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.069139 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.071605 (+ 2466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.071611 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.071787 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.071819 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:05:00.071911 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:05:00.072041 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15229,"mutex_wait_us":219,"thread_start_us":15275,"threads_started":1}
I20260620 15:05:00.073650 21550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.048975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33827 (local address 127.0.0.1:37450)
0620 15:05:00.063362 (+ 14387us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:00.063395 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:00.063517 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:00.066495 (+ 2978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:00.066502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:00.066537 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:00.067206 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.067220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.068968 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.068975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.071010 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.071024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.072848 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.072854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.073036 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.073066 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 15:05:00.073146 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 15:05:00.073271 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14269,"mutex_wait_us":148,"thread_start_us":14804,"threads_started":1}
I20260620 15:05:00.074054 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.049239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37450 (local address 127.2.70.1:33827)
0620 15:05:00.065934 (+ 16695us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.065941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.065969 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:05:00.066056 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.066063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.066281 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.066474 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:00.067430 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.068838 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.071235 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.072649 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.072805 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.073153 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.073236 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.073454 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16514,"thread_start_us":17009,"threads_started":1}
I20260620 15:05:00.076478 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.044820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35284 (local address 127.2.70.3:37651)
0620 15:05:00.062488 (+ 17668us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.062497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.062531 (+ 34us) server_negotiation.cc:407] Connection header received
0620 15:05:00.062617 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.062624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.062846 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.063010 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:00.064467 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.065977 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.069326 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.070765 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.070892 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.071159 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.075911 (+ 4752us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.076085 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17490,"thread_start_us":17310,"threads_started":1}
I20260620 15:05:00.076037 21339 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0968ba83aa42228defd53c2ba183b3 (DEFAULT_TABLE table=test-table [id=5036f303ed39498fbf98bcc9ae474c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:00.078027 21339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0968ba83aa42228defd53c2ba183b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:00.077956 21208 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0968ba83aa42228defd53c2ba183b3 (DEFAULT_TABLE table=test-table [id=5036f303ed39498fbf98bcc9ae474c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:00.079353 21208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0968ba83aa42228defd53c2ba183b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:00.079816 21469 tablet_service.cc:1467] Processing CreateTablet for tablet 1f0968ba83aa42228defd53c2ba183b3 (DEFAULT_TABLE table=test-table [id=5036f303ed39498fbf98bcc9ae474c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:00.081163 21469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f0968ba83aa42228defd53c2ba183b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:00.085597 21273 heartbeater.cc:503] Master 127.2.70.62:46697 was elected leader, sending a full tablet report...
I20260620 15:05:00.091611 21552 tablet_bootstrap.cc:492] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Bootstrap starting.
I20260620 15:05:00.093705 21553 tablet_bootstrap.cc:492] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: Bootstrap starting.
I20260620 15:05:00.095017 21554 tablet_bootstrap.cc:492] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Bootstrap starting.
I20260620 15:05:00.095942 21552 tablet_bootstrap.cc:654] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:00.097103 21553 tablet_bootstrap.cc:654] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:00.097234 21552 log.cc:824] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:00.098271 21553 log.cc:824] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:00.098857 21554 tablet_bootstrap.cc:654] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:00.099915 21554 log.cc:824] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:00.100618 21552 tablet_bootstrap.cc:492] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: No bootstrap required, opened a new log
I20260620 15:05:00.100900 21552 ts_tablet_manager.cc:1397] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260620 15:05:00.101157 21553 tablet_bootstrap.cc:492] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: No bootstrap required, opened a new log
I20260620 15:05:00.101446 21553 ts_tablet_manager.cc:1397] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260620 15:05:00.102773 21554 tablet_bootstrap.cc:492] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: No bootstrap required, opened a new log
I20260620 15:05:00.103061 21554 ts_tablet_manager.cc:1397] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260620 15:05:00.110864 21554 raft_consensus.cc:348] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.111327 21554 raft_consensus.cc:374] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:00.111441 21554 raft_consensus.cc:729] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020ff46557b1410bb703634412617f32, State: Initialized, Role: FOLLOWER
I20260620 15:05:00.111781 21552 raft_consensus.cc:348] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.111853 21553 raft_consensus.cc:348] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.112254 21553 raft_consensus.cc:374] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:00.112254 21552 raft_consensus.cc:374] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:00.112149 21554 consensus_queue.cc:260] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.112368 21552 raft_consensus.cc:729] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee19c9cfe564330bc0da3f4b14eb519, State: Initialized, Role: FOLLOWER
I20260620 15:05:00.112372 21553 raft_consensus.cc:729] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a998701b12a4ce6872a098ef7982b13, State: Initialized, Role: FOLLOWER
I20260620 15:05:00.112957 21553 consensus_queue.cc:260] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.112949 21552 consensus_queue.cc:260] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.114207 21404 heartbeater.cc:503] Master 127.2.70.62:46697 was elected leader, sending a full tablet report...
I20260620 15:05:00.115096 21552 ts_tablet_manager.cc:1428] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Time spent starting tablet: real 0.014s user 0.009s sys 0.006s
I20260620 15:05:00.116046 21553 ts_tablet_manager.cc:1428] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260620 15:05:00.117142 21534 heartbeater.cc:503] Master 127.2.70.62:46697 was elected leader, sending a full tablet report...
I20260620 15:05:00.118008 21554 ts_tablet_manager.cc:1428] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Time spent starting tablet: real 0.015s user 0.005s sys 0.008s
I20260620 15:05:00.193737 21558 raft_consensus.cc:482] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:05:00.194008 21558 raft_consensus.cc:504] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.195647 21558 leader_election.cc:288] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ee19c9cfe564330bc0da3f4b14eb519 (127.2.70.2:46809), 5a998701b12a4ce6872a098ef7982b13 (127.2.70.1:33827)
I20260620 15:05:00.202100 21536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.196000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:46809 (local address 127.2.70.3:33879)
0620 15:05:00.196364 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:00.196424 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:00.196596 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:00.197036 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:00.197040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:00.197081 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:00.197276 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.197290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.198504 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.198510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.200068 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.200081 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.201292 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.201298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.201517 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.201553 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 15:05:00.201635 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 15:05:00.201783 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":1}
I20260620 15:05:00.202113 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.196113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:33879 (local address 127.2.70.2:46809)
0620 15:05:00.196314 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.196320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.196538 (+ 218us) server_negotiation.cc:407] Connection header received
0620 15:05:00.196763 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.196770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.196865 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.196972 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:00.197449 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.198359 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.200235 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.201115 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.201227 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.201296 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.201693 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.201821 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:05:00.202491 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.196113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:34315 (local address 127.2.70.1:33827)
0620 15:05:00.196314 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.196320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.196983 (+ 663us) server_negotiation.cc:407] Connection header received
0620 15:05:00.197141 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.197146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.197233 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.197334 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:00.197941 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.198749 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.200545 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.201532 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.201664 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.201754 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.202099 (+ 345us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.202244 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:05:00.202791 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.196042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33827 (local address 127.2.70.3:34315)
0620 15:05:00.196852 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:00.196869 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:00.197000 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:00.197350 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:00.197376 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:00.197399 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:00.197764 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.197776 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.198906 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.198910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.200354 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:00.200367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.201719 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.201725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:00.201912 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.201944 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:05:00.202034 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:05:00.202166 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":459,"threads_started":1}
I20260620 15:05:00.203181 21359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0968ba83aa42228defd53c2ba183b3" candidate_uuid: "020ff46557b1410bb703634412617f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" is_pre_election: true
I20260620 15:05:00.203773 21359 raft_consensus.cc:2454] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 020ff46557b1410bb703634412617f32 in term 0.
I20260620 15:05:00.203945 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0968ba83aa42228defd53c2ba183b3" candidate_uuid: "020ff46557b1410bb703634412617f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a998701b12a4ce6872a098ef7982b13" is_pre_election: true
I20260620 15:05:00.204607 21228 raft_consensus.cc:2454] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 020ff46557b1410bb703634412617f32 in term 0.
I20260620 15:05:00.204574 21423 leader_election.cc:302] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 020ff46557b1410bb703634412617f32, 1ee19c9cfe564330bc0da3f4b14eb519; no voters:
I20260620 15:05:00.205084 21558 raft_consensus.cc:2781] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:05:00.205250 21558 raft_consensus.cc:482] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:05:00.205415 21558 raft_consensus.cc:3037] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:00.207731 21558 raft_consensus.cc:504] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.208333 21558 leader_election.cc:288] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [CANDIDATE]: Term 1 election: Requested vote from peers 1ee19c9cfe564330bc0da3f4b14eb519 (127.2.70.2:46809), 5a998701b12a4ce6872a098ef7982b13 (127.2.70.1:33827)
I20260620 15:05:00.208909 21228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0968ba83aa42228defd53c2ba183b3" candidate_uuid: "020ff46557b1410bb703634412617f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a998701b12a4ce6872a098ef7982b13"
I20260620 15:05:00.208899 21359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f0968ba83aa42228defd53c2ba183b3" candidate_uuid: "020ff46557b1410bb703634412617f32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ee19c9cfe564330bc0da3f4b14eb519"
I20260620 15:05:00.209128 21228 raft_consensus.cc:3037] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:00.209125 21359 raft_consensus.cc:3037] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:00.212555 21359 raft_consensus.cc:2454] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 020ff46557b1410bb703634412617f32 in term 1.
I20260620 15:05:00.212550 21228 raft_consensus.cc:2454] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 020ff46557b1410bb703634412617f32 in term 1.
I20260620 15:05:00.213032 21425 leader_election.cc:302] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 020ff46557b1410bb703634412617f32, 5a998701b12a4ce6872a098ef7982b13; no voters:
I20260620 15:05:00.213420 21558 raft_consensus.cc:2781] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:00.213948 21558 raft_consensus.cc:686] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 1 LEADER]: Becoming Leader. State: Replica: 020ff46557b1410bb703634412617f32, State: Running, Role: LEADER
I20260620 15:05:00.214442 21558 consensus_queue.cc:237] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } }
I20260620 15:05:00.219333 21087 catalog_manager.cc:5495] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 020ff46557b1410bb703634412617f32 (127.2.70.3). New cstate: current_term: 1 leader_uuid: "020ff46557b1410bb703634412617f32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020ff46557b1410bb703634412617f32" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 37651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 } health_report { overall_health: UNKNOWN } } }
W20260620 15:05:00.241097 21535 tablet.cc:2369] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:00.269160 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.263479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35298 (local address 127.2.70.3:37651)
0620 15:05:00.263677 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.263684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.263748 (+ 64us) server_negotiation.cc:407] Connection header received
0620 15:05:00.263984 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.263988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.264064 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.264223 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:00.264650 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.265390 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.266881 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.267303 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.267446 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.268035 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:00.268538 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:00.268727 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.268888 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.268932 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:05:00.277132 21228 raft_consensus.cc:1261] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 1 FOLLOWER]: Refusing update from remote peer 020ff46557b1410bb703634412617f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:00.277132 21359 raft_consensus.cc:1261] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 1 FOLLOWER]: Refusing update from remote peer 020ff46557b1410bb703634412617f32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260620 15:05:00.278028 21405 tablet.cc:2369] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:00.278183 21562 consensus_queue.cc:1035] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a998701b12a4ce6872a098ef7982b13" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:05:00.278617 21558 consensus_queue.cc:1035] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ee19c9cfe564330bc0da3f4b14eb519" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:05:00.291380 21567 mvcc.cc:205] Tried to move back new op lower bound from 7298940519525736448 to 7298940519282388992. Current Snapshot: MvccSnapshot[applied={T|T < 7298940519525736448}]
I20260620 15:05:00.303211 21087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:05:00.303503 21087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:05:00.306792 21087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00633dbaaaa948b08d4a4122bac328b6: Sending DeleteTablet for 3 replicas of tablet 1f0968ba83aa42228defd53c2ba183b3
I20260620 15:05:00.308005 21339 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0968ba83aa42228defd53c2ba183b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:00 UTC) from {username='slave'} at 127.0.0.1:43042
I20260620 15:05:00.308079 21469 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0968ba83aa42228defd53c2ba183b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:00 UTC) from {username='slave'} at 127.0.0.1:35284
I20260620 15:05:00.308151 21208 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f0968ba83aa42228defd53c2ba183b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:00 UTC) from {username='slave'} at 127.0.0.1:37450
I20260620 15:05:00.310082 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 21146
I20260620 15:05:00.320374 21577 tablet_replica.cc:330] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: stopping tablet replica
I20260620 15:05:00.321436 21576 tablet_replica.cc:330] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: stopping tablet replica
I20260620 15:05:00.321842 21576 raft_consensus.cc:2229] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:05:00.322508 21576 raft_consensus.cc:2258] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:00.320856 21577 raft_consensus.cc:2229] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:05:00.323251 21577 raft_consensus.cc:2258] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:00.324976 21575 tablet_replica.cc:330] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: stopping tablet replica
I20260620 15:05:00.325352 21577 ts_tablet_manager.cc:1905] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:00.325605 21575 raft_consensus.cc:2229] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:05:00.326171 21575 raft_consensus.cc:2258] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:00.328455 21576 ts_tablet_manager.cc:1905] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:00.329942 21575 ts_tablet_manager.cc:1905] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:00.333122 21577 ts_tablet_manager.cc:1918] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:00.333335 21577 log.cc:1196] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/1f0968ba83aa42228defd53c2ba183b3
I20260620 15:05:00.333922 21577 ts_tablet_manager.cc:1939] T 1f0968ba83aa42228defd53c2ba183b3 P 5a998701b12a4ce6872a098ef7982b13: Deleting consensus metadata
I20260620 15:05:00.336086 21074 catalog_manager.cc:4841] TS 5a998701b12a4ce6872a098ef7982b13 (127.2.70.1:33827): tablet 1f0968ba83aa42228defd53c2ba183b3 (table test-table [id=5036f303ed39498fbf98bcc9ae474c9f]) successfully deleted
I20260620 15:05:00.336890 21576 ts_tablet_manager.cc:1918] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:00.336881 21575 ts_tablet_manager.cc:1918] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:00.337047 21575 log.cc:1196] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/1f0968ba83aa42228defd53c2ba183b3
I20260620 15:05:00.337121 21576 log.cc:1196] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/1f0968ba83aa42228defd53c2ba183b3
I20260620 15:05:00.337672 21575 ts_tablet_manager.cc:1939] T 1f0968ba83aa42228defd53c2ba183b3 P 1ee19c9cfe564330bc0da3f4b14eb519: Deleting consensus metadata
I20260620 15:05:00.337747 21576 ts_tablet_manager.cc:1939] T 1f0968ba83aa42228defd53c2ba183b3 P 020ff46557b1410bb703634412617f32: Deleting consensus metadata
I20260620 15:05:00.338187 21551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.311566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37464 (local address 127.2.70.1:33827)
0620 15:05:00.311787 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.311810 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.318045 (+ 6235us) server_negotiation.cc:407] Connection header received
0620 15:05:00.318271 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.318279 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.318417 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.318546 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:00.320374 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.321891 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.324515 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.325165 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.325266 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.327419 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:00.327432 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:00.327453 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:00.328232 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:00.328297 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:00.328335 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:00.328424 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.337655 (+ 9231us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.337802 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 15:05:00.339406 21073 catalog_manager.cc:4841] TS 020ff46557b1410bb703634412617f32 (127.2.70.3:37651): tablet 1f0968ba83aa42228defd53c2ba183b3 (table test-table [id=5036f303ed39498fbf98bcc9ae474c9f]) successfully deleted
I20260620 15:05:00.339644 21268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:00.339601 21072 catalog_manager.cc:4841] TS 1ee19c9cfe564330bc0da3f4b14eb519 (127.2.70.2:46809): tablet 1f0968ba83aa42228defd53c2ba183b3 (table test-table [id=5036f303ed39498fbf98bcc9ae474c9f]) successfully deleted
I20260620 15:05:00.450783 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 21146
I20260620 15:05:00.470144 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 21277
I20260620 15:05:00.477890 21548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.471038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.2.70.2:46809)
0620 15:05:00.471194 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.471198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.471217 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:05:00.471406 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.471410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.471509 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.471591 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:00.472042 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.472802 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.474684 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.475084 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.475188 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.476356 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:00.476364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:00.476379 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:00.476956 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:00.477006 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:00.477026 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:00.477111 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.477297 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.477402 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 15:05:00.478623 21399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:00.581848 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 21277
I20260620 15:05:00.604660 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 21407
I20260620 15:05:00.612661 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:00.605465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35312 (local address 127.2.70.3:37651)
0620 15:05:00.605659 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:00.605663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:00.605701 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:05:00.605922 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:00.605926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:00.606048 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:00.606129 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:00.606625 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.607401 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.609265 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:00.609872 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:00.609977 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:00.611217 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:00.611225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:00.611240 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:00.611853 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:00.611911 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:00.611936 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:00.612016 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:00.612194 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 15:05:00.612274 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260620 15:05:00.613411 21529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:00.734400 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 21407
I20260620 15:05:00.754091 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 21056
I20260620 15:05:00.755234 21117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:00.862957 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 21056
2026-06-20T15:05:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2866 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:00 dist-test-slave-n5vv krb5kdc[21585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:00 dist-test-slave-n5vv krb5kdc[21585](info): set up 2 sockets
Jun 20 15:05:00 dist-test-slave-n5vv krb5kdc[21585](info): commencing operation
krb5kdc: starting...
W20260620 15:05:01.978150 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:02 dist-test-slave-n5vv krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:02.226428 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34651
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:02.560063 21598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:02.560349 21598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:02.560441 21598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:02.567657 21598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:02.567773 21598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:02.567821 21598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:02.567876 21598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:02.567914 21598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:02.577755 21598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:02.579131 21598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:02.580375 21598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:02.592244 21603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:02.592334 21604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:02.592365 21606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:02.593541 21598 server_base.cc:1034] running on GCE node
I20260620 15:05:02.594342 21598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:02.595775 21598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:02.597005 21598 hybrid_clock.cc:648] HybridClock initialized: now 1781967902596965 us; error 54 us; skew 500 ppm
I20260620 15:05:02.597349 21598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:02 dist-test-slave-n5vv krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967902, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:02.605168 21598 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:02.606730 21598 webserver.cc:460] Webserver started at http://127.2.70.62:37563/ using document root <none> and password file <none>
I20260620 15:05:02.607390 21598 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:02.607486 21598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:02.607723 21598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:02.610670 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "ffcb5f83001e4fefab2f1929e7abac45"
format_stamp: "Formatted at 2026-06-20 15:05:02 on dist-test-slave-n5vv"
server_key: "feeb09f38aa86fe2b5cea469127a0f86"
server_key_iv: "04a7a72254f6d2bb0ec14d515d1de59d"
server_key_version: "encryptionkey@0"
I20260620 15:05:02.611491 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "ffcb5f83001e4fefab2f1929e7abac45"
format_stamp: "Formatted at 2026-06-20 15:05:02 on dist-test-slave-n5vv"
server_key: "feeb09f38aa86fe2b5cea469127a0f86"
server_key_iv: "04a7a72254f6d2bb0ec14d515d1de59d"
server_key_version: "encryptionkey@0"
I20260620 15:05:02.616580 21598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:05:02.619867 21613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:02.621163 21598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:05:02.621315 21598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "ffcb5f83001e4fefab2f1929e7abac45"
format_stamp: "Formatted at 2026-06-20 15:05:02 on dist-test-slave-n5vv"
server_key: "feeb09f38aa86fe2b5cea469127a0f86"
server_key_iv: "04a7a72254f6d2bb0ec14d515d1de59d"
server_key_version: "encryptionkey@0"
I20260620 15:05:02.621539 21598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:02.635100 21598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:02.638831 21598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:02.639117 21598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:02.657253 21598 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:34651
I20260620 15:05:02.657286 21664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:34651 every 8 connection(s)
I20260620 15:05:02.658619 21598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:02.659940 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 21598
I20260620 15:05:02.660120 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:02.660647 2328 external_mini_cluster.cc:1428] Setting key d4c123d9a08245c89fe48e43385025ac
I20260620 15:05:02.663226 21665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:05:02 dist-test-slave-n5vv krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:02.676717 21665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45: Bootstrap starting.
I20260620 15:05:02.680286 21665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:02.681386 21665 log.cc:824] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:02.683877 21668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:02.662547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.2.70.62:34651)
0620 15:05:02.663331 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:02.663346 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:02.663400 (+ 54us) server_negotiation.cc:407] Connection header received
0620 15:05:02.665012 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:02.665033 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:02.665407 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:02.665802 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:02.667570 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:02.668915 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:02.670773 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:02.671482 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:02.671614 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:02.676477 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:02.676486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:02.676501 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:02.676540 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:02.679642 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:02.680358 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:02.680365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:02.680370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:02.680591 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:02.680989 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:02.680994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:02.680998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:02.681618 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:02.681933 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:02.682428 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 15:05:02.682911 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":364,"threads_started":1}
I20260620 15:05:02.684386 21665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45: No bootstrap required, opened a new log
I20260620 15:05:02.691584 21665 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcb5f83001e4fefab2f1929e7abac45" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34651 } }
I20260620 15:05:02.691968 21665 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:02.692075 21665 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffcb5f83001e4fefab2f1929e7abac45, State: Initialized, Role: FOLLOWER
I20260620 15:05:02.692724 21665 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcb5f83001e4fefab2f1929e7abac45" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34651 } }
I20260620 15:05:02.693007 21665 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:02.693158 21665 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:02.693336 21665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:02.695587 21665 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcb5f83001e4fefab2f1929e7abac45" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34651 } }
I20260620 15:05:02.696107 21665 leader_election.cc:302] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffcb5f83001e4fefab2f1929e7abac45; no voters:
I20260620 15:05:02.696707 21665 leader_election.cc:288] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:02.696844 21670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:02.697614 21670 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [term 1 LEADER]: Becoming Leader. State: Replica: ffcb5f83001e4fefab2f1929e7abac45, State: Running, Role: LEADER
I20260620 15:05:02.698112 21670 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcb5f83001e4fefab2f1929e7abac45" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34651 } }
I20260620 15:05:02.698922 21665 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:02.700799 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffcb5f83001e4fefab2f1929e7abac45. Latest consensus state: current_term: 1 leader_uuid: "ffcb5f83001e4fefab2f1929e7abac45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcb5f83001e4fefab2f1929e7abac45" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34651 } } }
I20260620 15:05:02.701196 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:02.702253 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffcb5f83001e4fefab2f1929e7abac45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcb5f83001e4fefab2f1929e7abac45" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34651 } } }
I20260620 15:05:02.702582 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:02.703711 21675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:02.708730 21675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:02.716759 21675 catalog_manager.cc:1347] Generated new cluster ID: bb008d4e5ee94c7086162bb3563f1b71
I20260620 15:05:02.716871 21675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:02.737407 21675 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:02.738792 21675 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:02.749198 21675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45: Generated new TSK 0
I20260620 15:05:02.750370 21675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.70.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
Entry for principal kudu/127.2.70.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab.
I20260620 15:05:02.821223 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34651
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:03.154449 21691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:03.154721 21691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:03.154888 21691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:03.162237 21691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:03.162366 21691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:03.162551 21691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:05:03.172164 21691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.1.keytab
--principal=kudu/127.2.70.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:03.173758 21691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:03.175261 21691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:03.190596 21697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:03.190677 21696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:03.190673 21699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:03.191746 21691 server_base.cc:1034] running on GCE node
I20260620 15:05:03.192343 21691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:03.193318 21691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:03.194540 21691 hybrid_clock.cc:648] HybridClock initialized: now 1781967903194485 us; error 68 us; skew 500 ppm
I20260620 15:05:03.194846 21691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:03 dist-test-slave-n5vv krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:03.197930 21691 init.cc:377] Logged in from keytab as kudu/127.2.70.1@KRBTEST.COM (short username kudu)
I20260620 15:05:03.199239 21691 webserver.cc:460] Webserver started at http://127.2.70.1:45949/ using document root <none> and password file <none>
I20260620 15:05:03.199939 21691 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:03.200073 21691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:03.200372 21691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:03.203352 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "5c0044b7adf641d5a722685ec25d4b1e"
format_stamp: "Formatted at 2026-06-20 15:05:03 on dist-test-slave-n5vv"
server_key: "16a3a6d156e84ed90456bdf5b86f99e9"
server_key_iv: "0c48dc8b0d5b1198b6374d46cf0d582c"
server_key_version: "encryptionkey@0"
I20260620 15:05:03.204146 21691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "5c0044b7adf641d5a722685ec25d4b1e"
format_stamp: "Formatted at 2026-06-20 15:05:03 on dist-test-slave-n5vv"
server_key: "16a3a6d156e84ed90456bdf5b86f99e9"
server_key_iv: "0c48dc8b0d5b1198b6374d46cf0d582c"
server_key_version: "encryptionkey@0"
I20260620 15:05:03.209133 21691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:05:03.212571 21706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:03.213843 21691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:05:03.214015 21691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "5c0044b7adf641d5a722685ec25d4b1e"
format_stamp: "Formatted at 2026-06-20 15:05:03 on dist-test-slave-n5vv"
server_key: "16a3a6d156e84ed90456bdf5b86f99e9"
server_key_iv: "0c48dc8b0d5b1198b6374d46cf0d582c"
server_key_version: "encryptionkey@0"
I20260620 15:05:03.214207 21691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:03.226393 21691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:03.230159 21691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:03.230445 21691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:03.231395 21691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:05:03.232695 21691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:05:03.232765 21691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:03.232939 21691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:05:03.233001 21691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:03.267172 21691 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:33503
I20260620 15:05:03.267207 21818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:33503 every 8 connection(s)
I20260620 15:05:03.268476 21691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:05:03.278337 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 21691
I20260620 15:05:03.278514 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:05:03.278949 2328 external_mini_cluster.cc:1428] Setting key 3c898cfb7cc264f32e7c97df9245b3c3
Jun 20 15:05:03 dist-test-slave-n5vv krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.1@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:03.288151 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:03.271009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:40699 (local address 127.2.70.62:34651)
0620 15:05:03.271436 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:03.271441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:03.272332 (+ 891us) server_negotiation.cc:407] Connection header received
0620 15:05:03.274201 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:03.274207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:03.274303 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:03.274425 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:03.275858 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:03.276544 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:03.278146 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:03.278645 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:03.278777 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:03.282499 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:03.282505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:03.282508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:03.282539 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:03.285109 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:03.285892 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:03.285900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:03.285908 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:03.286025 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:03.286446 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:03.286451 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:03.286454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:03.286712 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:03.286862 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:03.287564 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 15:05:03.287790 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":224,"threads_started":1}
I20260620 15:05:03.289068 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:03.271457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34651 (local address 127.2.70.1:40699)
0620 15:05:03.272175 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:03.272212 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:03.273924 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:03.274649 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:03.274657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:03.275143 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:03.275696 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:03.275712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:03.276693 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:03.276700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:03.277995 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:03.278006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:03.278870 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:03.278878 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:03.279157 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:03.279851 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:03.279880 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:03.282285 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:03.285255 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:03.285263 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:03.285279 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:03.285772 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:03.286133 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:03.286139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:03.286143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:03.286326 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:03.286846 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:03.286854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:03.287196 (+ 342us) client_negotiation.cc:769] Sending connection context
0620 15:05:03.287450 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 15:05:03.287990 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":355,"threads_started":1}
I20260620 15:05:03.290652 21819 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34651
I20260620 15:05:03.290953 21819 heartbeater.cc:464] Registering TS with master...
I20260620 15:05:03.291653 21819 heartbeater.cc:511] Master 127.2.70.62:34651 requested a full tablet report, sending...
I20260620 15:05:03.293747 21630 ts_manager.cc:194] Registered new tserver with Master: 5c0044b7adf641d5a722685ec25d4b1e (127.2.70.1:33503)
I20260620 15:05:03.295202 21630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.1@KRBTEST.COM'} at 127.2.70.1:40699
WARNING: no policy specified for kudu/127.2.70.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
Entry for principal kudu/127.2.70.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab.
I20260620 15:05:03.333372 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34651
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:03.662685 21825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:03.662969 21825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:03.663129 21825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:03.671188 21825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:03.671301 21825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:03.671475 21825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:05:03.682076 21825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.2.keytab
--principal=kudu/127.2.70.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:03.683426 21825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:03.684792 21825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:03.698575 21830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:03.698534 21831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:03.698549 21833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:03.699379 21825 server_base.cc:1034] running on GCE node
I20260620 15:05:03.700063 21825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:03.701041 21825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:03.702266 21825 hybrid_clock.cc:648] HybridClock initialized: now 1781967903702201 us; error 59 us; skew 500 ppm
I20260620 15:05:03.702591 21825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:03 dist-test-slave-n5vv krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:03.705921 21825 init.cc:377] Logged in from keytab as kudu/127.2.70.2@KRBTEST.COM (short username kudu)
I20260620 15:05:03.707285 21825 webserver.cc:460] Webserver started at http://127.2.70.2:45849/ using document root <none> and password file <none>
I20260620 15:05:03.707984 21825 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:03.708123 21825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:03.708432 21825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:03.711345 21825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "ead9b05373a24a5a8799875fd1d46ae0"
format_stamp: "Formatted at 2026-06-20 15:05:03 on dist-test-slave-n5vv"
server_key: "10444915af682d28348b6838f97699e7"
server_key_iv: "7d29224f16193a301c3b152836a6a99b"
server_key_version: "encryptionkey@0"
I20260620 15:05:03.712136 21825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "ead9b05373a24a5a8799875fd1d46ae0"
format_stamp: "Formatted at 2026-06-20 15:05:03 on dist-test-slave-n5vv"
server_key: "10444915af682d28348b6838f97699e7"
server_key_iv: "7d29224f16193a301c3b152836a6a99b"
server_key_version: "encryptionkey@0"
I20260620 15:05:03.717469 21825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260620 15:05:03.721056 21840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:03.722455 21825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:05:03.722625 21825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "ead9b05373a24a5a8799875fd1d46ae0"
format_stamp: "Formatted at 2026-06-20 15:05:03 on dist-test-slave-n5vv"
server_key: "10444915af682d28348b6838f97699e7"
server_key_iv: "7d29224f16193a301c3b152836a6a99b"
server_key_version: "encryptionkey@0"
I20260620 15:05:03.722837 21825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:03.783885 21825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:03.787850 21825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:03.788169 21825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:03.789295 21825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:05:03.790771 21825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:05:03.790882 21825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:03.791028 21825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:05:03.791102 21825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:03.824041 21825 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:35061
I20260620 15:05:03.824090 21952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:35061 every 8 connection(s)
I20260620 15:05:03.825417 21825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:05:03.828333 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 21825
I20260620 15:05:03.828519 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:05:03.829002 2328 external_mini_cluster.cc:1428] Setting key 3a6e633f854207021ea14212d35cb3cd
Jun 20 15:05:03 dist-test-slave-n5vv krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967903, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.2@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:03.845788 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:03.828083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:41241 (local address 127.2.70.62:34651)
0620 15:05:03.828702 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:03.828709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:03.831607 (+ 2898us) server_negotiation.cc:407] Connection header received
0620 15:05:03.831659 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:03.831663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:03.831723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:03.831843 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:03.833364 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:03.834152 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:03.835710 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:03.836131 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:03.836260 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:03.839961 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:03.839966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:03.839970 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:03.840002 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:03.842597 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:03.843493 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:03.843497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:03.843500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:03.843616 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:03.844080 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:03.844083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:03.844085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:03.844327 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:03.844432 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:03.845280 (+ 848us) server_negotiation.cc:299] Negotiation successful
0620 15:05:03.845441 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":335,"threads_started":1}
I20260620 15:05:03.847303 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:03.828758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34651 (local address 127.2.70.2:41241)
0620 15:05:03.829592 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:03.829638 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:03.831325 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:03.832118 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:03.832126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:03.832626 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:03.833194 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:03.833210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:03.834315 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:03.834319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:03.835536 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:03.835547 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:03.836360 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:03.836366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:03.836598 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:03.837280 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:03.837301 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:03.839745 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:03.842733 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:03.842745 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:03.842764 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:03.843379 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:03.843718 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:03.843725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:03.843731 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:03.843980 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:03.844456 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:03.844467 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:03.844915 (+ 448us) client_negotiation.cc:769] Sending connection context
0620 15:05:03.845221 (+ 306us) client_negotiation.cc:240] Negotiation successful
0620 15:05:03.845852 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":348,"threads_started":1}
I20260620 15:05:03.848814 21953 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34651
I20260620 15:05:03.849172 21953 heartbeater.cc:464] Registering TS with master...
I20260620 15:05:03.850162 21953 heartbeater.cc:511] Master 127.2.70.62:34651 requested a full tablet report, sending...
I20260620 15:05:03.851689 21630 ts_manager.cc:194] Registered new tserver with Master: ead9b05373a24a5a8799875fd1d46ae0 (127.2.70.2:35061)
I20260620 15:05:03.852499 21630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.2@KRBTEST.COM'} at 127.2.70.2:41241
WARNING: no policy specified for kudu/127.2.70.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
Entry for principal kudu/127.2.70.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab.
I20260620 15:05:03.884204 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34651
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:04.208817 21959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:04.209105 21959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:04.209199 21959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:04.216655 21959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:04.216763 21959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:04.216876 21959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:05:04.226802 21959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.3.keytab
--principal=kudu/127.2.70.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:04.228091 21959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:04.229358 21959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:04.242446 21965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:04.242374 21964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:04.242480 21967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:04.243233 21959 server_base.cc:1034] running on GCE node
I20260620 15:05:04.243836 21959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:04.244819 21959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:04.246017 21959 hybrid_clock.cc:648] HybridClock initialized: now 1781967904245972 us; error 49 us; skew 500 ppm
I20260620 15:05:04.246373 21959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:04 dist-test-slave-n5vv krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967904, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:04.249522 21959 init.cc:377] Logged in from keytab as kudu/127.2.70.3@KRBTEST.COM (short username kudu)
I20260620 15:05:04.250982 21959 webserver.cc:460] Webserver started at http://127.2.70.3:42589/ using document root <none> and password file <none>
I20260620 15:05:04.251684 21959 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:04.251828 21959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:04.252138 21959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:04.255062 21959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "cd3b8ed1cab44ddda581b699f07e888e"
format_stamp: "Formatted at 2026-06-20 15:05:04 on dist-test-slave-n5vv"
server_key: "333b21684c1968f1f13ca3b69eeb7a33"
server_key_iv: "274221ae012faa66114083b9355a8cd8"
server_key_version: "encryptionkey@0"
I20260620 15:05:04.255836 21959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "cd3b8ed1cab44ddda581b699f07e888e"
format_stamp: "Formatted at 2026-06-20 15:05:04 on dist-test-slave-n5vv"
server_key: "333b21684c1968f1f13ca3b69eeb7a33"
server_key_iv: "274221ae012faa66114083b9355a8cd8"
server_key_version: "encryptionkey@0"
I20260620 15:05:04.261175 21959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260620 15:05:04.264560 21974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:04.265961 21959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:05:04.266135 21959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "cd3b8ed1cab44ddda581b699f07e888e"
format_stamp: "Formatted at 2026-06-20 15:05:04 on dist-test-slave-n5vv"
server_key: "333b21684c1968f1f13ca3b69eeb7a33"
server_key_iv: "274221ae012faa66114083b9355a8cd8"
server_key_version: "encryptionkey@0"
I20260620 15:05:04.266340 21959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:04.285341 21959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:04.289124 21959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:04.289422 21959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:04.290426 21959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:05:04.291697 21959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:05:04.291766 21959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:04.291945 21959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:05:04.291997 21959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:04.298153 21819 heartbeater.cc:503] Master 127.2.70.62:34651 was elected leader, sending a full tablet report...
I20260620 15:05:04.325635 21959 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:36991
I20260620 15:05:04.325670 22086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:36991 every 8 connection(s)
I20260620 15:05:04.326992 21959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:05:04.327831 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 21959
I20260620 15:05:04.328015 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:05:04.328461 2328 external_mini_cluster.cc:1428] Setting key 19110b42663342dbdb16899cb4c15019
Jun 20 15:05:04 dist-test-slave-n5vv krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967904, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.3@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:04.347790 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.331186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:47643 (local address 127.2.70.62:34651)
0620 15:05:04.331406 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.331412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.332682 (+ 1270us) server_negotiation.cc:407] Connection header received
0620 15:05:04.334540 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.334545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.334626 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.334712 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:04.336171 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.336897 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.338432 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.338826 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.338928 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.342356 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:04.342360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:04.342363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:04.342398 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:04.345241 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.345943 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.345947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.345949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.346060 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.346413 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.346416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.346418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.346663 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:04.346772 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.347379 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.347559 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:05:04.348812 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.331813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34651 (local address 127.2.70.3:47643)
0620 15:05:04.332528 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.332565 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:04.334277 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:04.334926 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:04.334933 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:04.335472 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:04.336028 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.336045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.337021 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.337025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.338313 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.338325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.339007 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.339012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.339192 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.339781 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:04.339806 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:04.341992 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:04.345358 (+ 3366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:04.345364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:04.345382 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:04.345841 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:04.346149 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:04.346152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:04.346154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:04.346318 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:04.346764 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:04.346769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:04.347053 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 15:05:04.347274 (+ 221us) client_negotiation.cc:240] Negotiation successful
0620 15:05:04.347633 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":277,"threads_started":1}
I20260620 15:05:04.350118 22087 heartbeater.cc:347] Connected to a master server at 127.2.70.62:34651
I20260620 15:05:04.350437 22087 heartbeater.cc:464] Registering TS with master...
I20260620 15:05:04.351171 22087 heartbeater.cc:511] Master 127.2.70.62:34651 requested a full tablet report, sending...
I20260620 15:05:04.352566 21630 ts_manager.cc:194] Registered new tserver with Master: cd3b8ed1cab44ddda581b699f07e888e (127.2.70.3:36991)
I20260620 15:05:04.353340 21630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.70.3@KRBTEST.COM'} at 127.2.70.3:47643
I20260620 15:05:04.355901 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:05:04.371973 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.359740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.2.70.62:34651)
0620 15:05:04.359891 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.359917 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.360309 (+ 392us) server_negotiation.cc:407] Connection header received
0620 15:05:04.360520 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.360524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.360581 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.360680 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:04.362108 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.362836 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.364092 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.364477 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.364606 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.366954 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:04.366960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:04.366963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:04.366992 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:04.369317 (+ 2325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.369984 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.369992 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.369995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.370124 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.370513 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.370518 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.370521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.370829 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:04.371092 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.371445 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.371663 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:05:04.375501 21630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:05:04.382210 21630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:05:04.418777 22101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.394449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.0.0.1:37046)
0620 15:05:04.399515 (+ 5066us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.399535 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:04.399644 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:04.411168 (+ 11524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:04.411173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:04.411201 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:04.411618 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.411632 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.413766 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.413774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.415753 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.415765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.417661 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.417670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.417877 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.417929 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 15:05:04.418064 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 15:05:04.418273 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4935,"mutex_wait_us":1,"thread_start_us":4809,"threads_started":1}
I20260620 15:05:04.419219 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.399847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:35061 (local address 127.0.0.1:40040)
0620 15:05:04.410846 (+ 10999us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.410865 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:04.411000 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:04.412143 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:04.412149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:04.412176 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:04.412431 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.412438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.413827 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.413839 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.416388 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.416403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.418272 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.418280 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.418473 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.418507 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 15:05:04.418608 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 15:05:04.418734 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10884,"mutex_wait_us":154,"spinlock_wait_cycles":768,"thread_start_us":10741,"threads_started":1}
I20260620 15:05:04.419519 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.399903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40040 (local address 127.2.70.2:35061)
0620 15:05:04.411395 (+ 11492us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.411406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.411434 (+ 28us) server_negotiation.cc:407] Connection header received
0620 15:05:04.411536 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.411543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.411775 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.411978 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:04.412589 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.413678 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.416556 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.418073 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.418188 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.418379 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.418673 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.418876 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11341,"thread_start_us":12005,"threads_started":1}
I20260620 15:05:04.420020 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.393860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:36991 (local address 127.0.0.1:40026)
0620 15:05:04.394622 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.394670 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:04.394831 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:04.410779 (+ 15948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:04.410787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:04.410818 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:04.411286 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.411303 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.413677 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.413683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.416388 (+ 2705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.416403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.419325 (+ 2922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.419332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.419502 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.419524 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 15:05:04.419575 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 15:05:04.419732 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":222,"thread_start_us":739,"threads_started":1}
I20260620 15:05:04.420102 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.394657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.2.70.1:33503)
0620 15:05:04.410000 (+ 15343us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.410008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.410032 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:05:04.410120 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.410126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.410287 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.411817 (+ 1530us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:04.412088 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.413585 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.415919 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.417127 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.417242 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.417474 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.419612 (+ 2138us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.419785 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15175,"thread_start_us":16152,"threads_started":1}
I20260620 15:05:04.421707 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.394358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.2.70.3:36991)
0620 15:05:04.409831 (+ 15473us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.409841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.409879 (+ 38us) server_negotiation.cc:407] Connection header received
0620 15:05:04.409987 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.409996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.410288 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.411516 (+ 1228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:04.411790 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.413521 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.417594 (+ 4073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.419090 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.420260 (+ 1170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.420499 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.420553 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.420727 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15256,"thread_start_us":16076,"threads_started":1}
I20260620 15:05:04.423761 21754 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4b7ebb7823431985d1d537199d3e06 (DEFAULT_TABLE table=test-table [id=fe3cd59e2f964bdab40da5d04cd1b369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:04.423761 21888 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4b7ebb7823431985d1d537199d3e06 (DEFAULT_TABLE table=test-table [id=fe3cd59e2f964bdab40da5d04cd1b369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:04.424582 22022 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4b7ebb7823431985d1d537199d3e06 (DEFAULT_TABLE table=test-table [id=fe3cd59e2f964bdab40da5d04cd1b369]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:04.425387 21754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4b7ebb7823431985d1d537199d3e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:04.425961 21888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4b7ebb7823431985d1d537199d3e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:04.425962 22022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4b7ebb7823431985d1d537199d3e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:04.437162 22105 tablet_bootstrap.cc:492] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Bootstrap starting.
I20260620 15:05:04.440270 22106 tablet_bootstrap.cc:492] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Bootstrap starting.
I20260620 15:05:04.441156 22107 tablet_bootstrap.cc:492] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: Bootstrap starting.
I20260620 15:05:04.441460 22105 tablet_bootstrap.cc:654] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:04.442836 22105 log.cc:824] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:04.443749 22106 tablet_bootstrap.cc:654] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:04.444023 22107 tablet_bootstrap.cc:654] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:04.445065 22107 log.cc:824] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:04.445238 22106 log.cc:824] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:04.447856 22105 tablet_bootstrap.cc:492] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: No bootstrap required, opened a new log
I20260620 15:05:04.448173 22105 ts_tablet_manager.cc:1397] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260620 15:05:04.449738 22106 tablet_bootstrap.cc:492] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: No bootstrap required, opened a new log
I20260620 15:05:04.449980 22106 ts_tablet_manager.cc:1397] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260620 15:05:04.450076 22107 tablet_bootstrap.cc:492] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: No bootstrap required, opened a new log
I20260620 15:05:04.450287 22107 ts_tablet_manager.cc:1397] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20260620 15:05:04.457471 22107 raft_consensus.cc:348] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.458062 22107 raft_consensus.cc:374] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:04.458194 22107 raft_consensus.cc:729] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd3b8ed1cab44ddda581b699f07e888e, State: Initialized, Role: FOLLOWER
I20260620 15:05:04.458909 22107 consensus_queue.cc:260] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.459375 22105 raft_consensus.cc:348] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.459815 22105 raft_consensus.cc:374] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:04.459910 22105 raft_consensus.cc:729] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c0044b7adf641d5a722685ec25d4b1e, State: Initialized, Role: FOLLOWER
I20260620 15:05:04.460129 22087 heartbeater.cc:503] Master 127.2.70.62:34651 was elected leader, sending a full tablet report...
I20260620 15:05:04.460515 22105 consensus_queue.cc:260] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.460541 22106 raft_consensus.cc:348] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.460970 22106 raft_consensus.cc:374] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:04.461097 22106 raft_consensus.cc:729] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead9b05373a24a5a8799875fd1d46ae0, State: Initialized, Role: FOLLOWER
I20260620 15:05:04.462137 22107 ts_tablet_manager.cc:1428] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260620 15:05:04.461968 22106 consensus_queue.cc:260] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.464125 22105 ts_tablet_manager.cc:1428] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
I20260620 15:05:04.465126 21953 heartbeater.cc:503] Master 127.2.70.62:34651 was elected leader, sending a full tablet report...
I20260620 15:05:04.465814 22106 ts_tablet_manager.cc:1428] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260620 15:05:04.478518 22112 raft_consensus.cc:482] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:05:04.478816 22112 raft_consensus.cc:504] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.480480 22112 leader_election.cc:288] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd3b8ed1cab44ddda581b699f07e888e (127.2.70.3:36991), ead9b05373a24a5a8799875fd1d46ae0 (127.2.70.2:35061)
I20260620 15:05:04.487545 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.480961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:52957 (local address 127.2.70.3:36991)
0620 15:05:04.481201 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.481209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.482022 (+ 813us) server_negotiation.cc:407] Connection header received
0620 15:05:04.482235 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.482241 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.482318 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.482412 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:04.483111 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.483881 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.485744 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.486507 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.486602 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.486676 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.487153 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.487289 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:05:04.487636 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.480889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:59051 (local address 127.2.70.2:35061)
0620 15:05:04.481095 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.481099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.482256 (+ 1157us) server_negotiation.cc:407] Connection header received
0620 15:05:04.482451 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.482456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.482528 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.482624 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:04.483237 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.483966 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.485807 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.486548 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.486814 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.486906 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.487252 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.487399 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:05:04.487641 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.480995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:36991 (local address 127.2.70.1:52957)
0620 15:05:04.481886 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.481919 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:04.482084 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:04.482450 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:04.482454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:04.482480 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:04.482959 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.482973 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.484014 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.484022 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.485604 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.485616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.486821 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.486830 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.487008 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.487031 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:05:04.487104 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 15:05:04.487220 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":30,"thread_start_us":499,"threads_started":1}
I20260620 15:05:04.488093 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.480994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:35061 (local address 127.2.70.1:59051)
0620 15:05:04.482121 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.482155 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:04.482283 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:04.482647 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:04.482650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:04.482672 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:04.483065 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.483076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.484105 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.484109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.485634 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.485646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.486946 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.486952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.487071 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.487119 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 15:05:04.487209 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 15:05:04.487320 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":177,"thread_start_us":507,"threads_started":1}
I20260620 15:05:04.488281 22111 raft_consensus.cc:482] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:05:04.488731 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4b7ebb7823431985d1d537199d3e06" candidate_uuid: "5c0044b7adf641d5a722685ec25d4b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3b8ed1cab44ddda581b699f07e888e" is_pre_election: true
I20260620 15:05:04.488608 22111 raft_consensus.cc:504] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.489277 21908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4b7ebb7823431985d1d537199d3e06" candidate_uuid: "5c0044b7adf641d5a722685ec25d4b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead9b05373a24a5a8799875fd1d46ae0" is_pre_election: true
I20260620 15:05:04.489583 22042 raft_consensus.cc:2454] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c0044b7adf641d5a722685ec25d4b1e in term 0.
I20260620 15:05:04.489794 21908 raft_consensus.cc:2454] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c0044b7adf641d5a722685ec25d4b1e in term 0.
I20260620 15:05:04.490331 21708 leader_election.cc:302] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c0044b7adf641d5a722685ec25d4b1e, cd3b8ed1cab44ddda581b699f07e888e; no voters:
I20260620 15:05:04.490687 22111 leader_election.cc:288] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c0044b7adf641d5a722685ec25d4b1e (127.2.70.1:33503), ead9b05373a24a5a8799875fd1d46ae0 (127.2.70.2:35061)
I20260620 15:05:04.490763 22112 raft_consensus.cc:2781] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:05:04.490922 22112 raft_consensus.cc:482] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:05:04.491035 22112 raft_consensus.cc:3037] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:04.494449 22112 raft_consensus.cc:504] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.495546 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4b7ebb7823431985d1d537199d3e06" candidate_uuid: "5c0044b7adf641d5a722685ec25d4b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd3b8ed1cab44ddda581b699f07e888e"
I20260620 15:05:04.495572 21908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4b7ebb7823431985d1d537199d3e06" candidate_uuid: "5c0044b7adf641d5a722685ec25d4b1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead9b05373a24a5a8799875fd1d46ae0"
I20260620 15:05:04.495751 22042 raft_consensus.cc:3037] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:04.495764 21908 raft_consensus.cc:3037] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:04.497854 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.490907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.2.70.3:46387)
0620 15:05:04.491085 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.491104 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:04.491210 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:04.491943 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:04.491948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:04.491975 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:04.492125 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.492134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.493560 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.493567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.495318 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.495332 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.497064 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.497070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.497268 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.497300 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 15:05:04.497392 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 15:05:04.497533 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":5}
I20260620 15:05:04.498540 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.490962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:46387 (local address 127.2.70.1:33503)
0620 15:05:04.491146 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.491151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.491172 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:05:04.491605 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.491609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.491688 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.491778 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:04.492278 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.493345 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.495505 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.496732 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.496856 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.496972 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.498170 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.498308 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:05:04.499019 21908 raft_consensus.cc:2454] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c0044b7adf641d5a722685ec25d4b1e in term 1.
I20260620 15:05:04.499264 22112 leader_election.cc:288] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [CANDIDATE]: Term 1 election: Requested vote from peers cd3b8ed1cab44ddda581b699f07e888e (127.2.70.3:36991), ead9b05373a24a5a8799875fd1d46ae0 (127.2.70.2:35061)
I20260620 15:05:04.499312 21774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4b7ebb7823431985d1d537199d3e06" candidate_uuid: "cd3b8ed1cab44ddda581b699f07e888e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c0044b7adf641d5a722685ec25d4b1e" is_pre_election: true
I20260620 15:05:04.499552 21707 leader_election.cc:302] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c0044b7adf641d5a722685ec25d4b1e, ead9b05373a24a5a8799875fd1d46ae0; no voters:
I20260620 15:05:04.499588 22042 raft_consensus.cc:2454] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c0044b7adf641d5a722685ec25d4b1e in term 1.
I20260620 15:05:04.499898 21774 raft_consensus.cc:2379] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd3b8ed1cab44ddda581b699f07e888e in current term 1: Already voted for candidate 5c0044b7adf641d5a722685ec25d4b1e in this term.
I20260620 15:05:04.500236 22112 raft_consensus.cc:2781] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:04.500717 22112 raft_consensus.cc:686] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 1 LEADER]: Becoming Leader. State: Replica: 5c0044b7adf641d5a722685ec25d4b1e, State: Running, Role: LEADER
I20260620 15:05:04.501201 22112 consensus_queue.cc:237] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } }
I20260620 15:05:04.502015 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.491299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:35061 (local address 127.2.70.3:49165)
0620 15:05:04.492983 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.493001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:04.493103 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:04.493987 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:04.493992 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:04.494016 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:04.494475 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.494489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.495939 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.495944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.498137 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:04.498150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.501351 (+ 3201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.501355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:04.501537 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.501574 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 15:05:04.501646 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 15:05:04.501730 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":2,"thread_start_us":1335,"threads_started":1}
I20260620 15:05:04.502072 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.491485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:49165 (local address 127.2.70.2:35061)
0620 15:05:04.493627 (+ 2142us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.493633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.493652 (+ 19us) server_negotiation.cc:407] Connection header received
0620 15:05:04.493706 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.493711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.493773 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.493848 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:04.494648 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.495769 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.498294 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.501030 (+ 2736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.501113 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.501221 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.501698 (+ 477us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.501828 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2019}
I20260620 15:05:04.502779 21908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4b7ebb7823431985d1d537199d3e06" candidate_uuid: "cd3b8ed1cab44ddda581b699f07e888e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ead9b05373a24a5a8799875fd1d46ae0" is_pre_election: true
I20260620 15:05:04.503052 21908 raft_consensus.cc:2379] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd3b8ed1cab44ddda581b699f07e888e in current term 1: Already voted for candidate 5c0044b7adf641d5a722685ec25d4b1e in this term.
I20260620 15:05:04.503598 21975 leader_election.cc:302] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd3b8ed1cab44ddda581b699f07e888e; no voters: 5c0044b7adf641d5a722685ec25d4b1e, ead9b05373a24a5a8799875fd1d46ae0
I20260620 15:05:04.503983 22111 raft_consensus.cc:2726] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 15:05:04.505888 21629 catalog_manager.cc:5495] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c0044b7adf641d5a722685ec25d4b1e (127.2.70.1). New cstate: current_term: 1 leader_uuid: "5c0044b7adf641d5a722685ec25d4b1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c0044b7adf641d5a722685ec25d4b1e" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 } health_report { overall_health: UNKNOWN } } }
W20260620 15:05:04.520954 21820 tablet.cc:2369] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:04.558560 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.553256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.2.70.1:33503)
0620 15:05:04.553521 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.553528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.553574 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:05:04.553758 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.553761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.553861 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.553998 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:04.554302 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.555047 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.556383 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.556775 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.556860 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.557359 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:04.557949 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:04.558058 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.558169 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.558231 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 15:05:04.566334 21908 raft_consensus.cc:1261] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 1 FOLLOWER]: Refusing update from remote peer 5c0044b7adf641d5a722685ec25d4b1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:04.566390 22042 raft_consensus.cc:1261] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 1 FOLLOWER]: Refusing update from remote peer 5c0044b7adf641d5a722685ec25d4b1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:04.567183 22112 consensus_queue.cc:1035] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ead9b05373a24a5a8799875fd1d46ae0" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:05:04.567478 22117 consensus_queue.cc:1035] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd3b8ed1cab44ddda581b699f07e888e" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:05:04.577458 21954 tablet.cc:2369] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:04.580063 22122 mvcc.cc:205] Tried to move back new op lower bound from 7298940537094406144 to 7298940536841220096. Current Snapshot: MvccSnapshot[applied={T|T < 7298940537094406144}]
I20260620 15:05:04.580168 22120 mvcc.cc:205] Tried to move back new op lower bound from 7298940537094406144 to 7298940536841220096. Current Snapshot: MvccSnapshot[applied={T|T < 7298940537094406144}]
I20260620 15:05:04.592727 21630 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:05:04.593022 21630 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:05:04.596279 21630 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffcb5f83001e4fefab2f1929e7abac45: Sending DeleteTablet for 3 replicas of tablet 3a4b7ebb7823431985d1d537199d3e06
I20260620 15:05:04.597414 22022 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4b7ebb7823431985d1d537199d3e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:04 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:40026
I20260620 15:05:04.597524 21888 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4b7ebb7823431985d1d537199d3e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:04 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:40040
I20260620 15:05:04.598313 21754 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4b7ebb7823431985d1d537199d3e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:04 UTC) from {username='kudu', principal='kudu/127.2.70.62@KRBTEST.COM'} at 127.0.0.1:37046
I20260620 15:05:04.598773 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 21691
I20260620 15:05:04.610248 22129 tablet_replica.cc:330] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: stopping tablet replica
I20260620 15:05:04.610764 22129 raft_consensus.cc:2229] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:05:04.611291 22129 raft_consensus.cc:2258] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:04.612061 22130 tablet_replica.cc:330] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: stopping tablet replica
I20260620 15:05:04.612519 22130 raft_consensus.cc:2229] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:05:04.613016 22130 raft_consensus.cc:2258] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:04.615173 22130 ts_tablet_manager.cc:1905] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:04.617170 22131 tablet_replica.cc:330] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: stopping tablet replica
I20260620 15:05:04.617631 22131 raft_consensus.cc:2229] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:05:04.619532 22129 ts_tablet_manager.cc:1905] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:04.622619 22130 ts_tablet_manager.cc:1918] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:04.622781 22130 log.cc:1196] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/3a4b7ebb7823431985d1d537199d3e06
Jun 20 15:05:04 dist-test-slave-n5vv krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.1@KRBTEST.COM
I20260620 15:05:04.623345 22130 ts_tablet_manager.cc:1939] T 3a4b7ebb7823431985d1d537199d3e06 P ead9b05373a24a5a8799875fd1d46ae0: Deleting consensus metadata
I20260620 15:05:04.623912 22131 raft_consensus.cc:2258] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:04.625347 21614 catalog_manager.cc:4841] TS ead9b05373a24a5a8799875fd1d46ae0 (127.2.70.2:35061): tablet 3a4b7ebb7823431985d1d537199d3e06 (table test-table [id=fe3cd59e2f964bdab40da5d04cd1b369]) successfully deleted
I20260620 15:05:04.626561 22131 ts_tablet_manager.cc:1905] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:04.627522 22129 ts_tablet_manager.cc:1918] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:04.627761 22129 log.cc:1196] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/3a4b7ebb7823431985d1d537199d3e06
I20260620 15:05:04.628332 22129 ts_tablet_manager.cc:1939] T 3a4b7ebb7823431985d1d537199d3e06 P cd3b8ed1cab44ddda581b699f07e888e: Deleting consensus metadata
I20260620 15:05:04.631990 21615 catalog_manager.cc:4841] TS cd3b8ed1cab44ddda581b699f07e888e (127.2.70.3:36991): tablet 3a4b7ebb7823431985d1d537199d3e06 (table test-table [id=fe3cd59e2f964bdab40da5d04cd1b369]) successfully deleted
I20260620 15:05:04.635128 22131 ts_tablet_manager.cc:1918] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:04.635243 22102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.600236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37066 (local address 127.2.70.1:33503)
0620 15:05:04.601953 (+ 1717us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.601962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.610278 (+ 8316us) server_negotiation.cc:407] Connection header received
0620 15:05:04.610821 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.610826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.610931 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.611010 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:04.613555 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.614701 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.617918 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.618602 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.618724 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.627221 (+ 8497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:04.627241 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:04.627262 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:04.627330 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:04.631676 (+ 4346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.632756 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.632765 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.632769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.632962 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.633417 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.633422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.633425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.633992 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:04.634307 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.634627 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.634868 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1507}
I20260620 15:05:04.635327 22131 log.cc:1196] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/3a4b7ebb7823431985d1d537199d3e06
I20260620 15:05:04.636008 21814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:04.735090 22131 ts_tablet_manager.cc:1939] T 3a4b7ebb7823431985d1d537199d3e06 P 5c0044b7adf641d5a722685ec25d4b1e: Deleting consensus metadata
I20260620 15:05:04.735705 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 21691
W20260620 15:05:04.754832 21614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.70.1:33503 (error 108)
I20260620 15:05:04.755093 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 21825
I20260620 15:05:04.756839 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.756439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.0.0.1:37074)
0620 15:05:04.756605 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.756707 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:05:04.757066 21614 catalog_manager.cc:4584] TS 5c0044b7adf641d5a722685ec25d4b1e (127.2.70.1:33503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3a4b7ebb7823431985d1d537199d3e06: Network error: Client connection negotiation failed: client connection to 127.2.70.1:33503: connect: Connection refused (error 111)
Jun 20 15:05:04 dist-test-slave-n5vv krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.2@KRBTEST.COM
I20260620 15:05:04.770285 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.756001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40056 (local address 127.2.70.2:35061)
0620 15:05:04.756183 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.756189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.756209 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:05:04.756418 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.756423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.756488 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.756614 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:04.758083 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.758786 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.760079 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.760474 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.760605 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.764840 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:04.764853 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:04.764868 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:04.764900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:04.767506 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.768203 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.768210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.768212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.768328 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.768691 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.768697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.768701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.769094 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:04.769405 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.769761 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.769982 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:05:04.770987 21948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:04.810611 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.810128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.0.0.1:37086)
0620 15:05:04.810352 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.810459 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 15:05:04.811646 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.811329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.0.0.1:37098)
0620 15:05:04.811459 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.811540 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 15:05:04.860844 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 21825
I20260620 15:05:04.872434 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.871764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.0.0.1:37110)
0620 15:05:04.872043 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.872199 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260620 15:05:04.873636 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.873158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.0.0.1:37122)
0620 15:05:04.873336 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.873447 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 15:05:04.878847 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 21959
Jun 20 15:05:04 dist-test-slave-n5vv krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.3@KRBTEST.COM
I20260620 15:05:04.893841 22100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.879535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.2.70.3:36991)
0620 15:05:04.879712 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:04.879719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:04.879740 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:05:04.879971 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:04.879975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:04.880038 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:04.880153 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:04.881440 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.882228 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.883471 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:04.883860 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:04.883987 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:04.888158 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:04.888173 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:04.888190 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:04.888224 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:04.890824 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.891491 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.891499 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.891501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.891629 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:04.892022 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:04.892027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:04.892030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:04.892408 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:04.892705 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:04.893045 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 15:05:04.893265 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:05:04.894481 22082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:04.954658 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.954053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.0.0.1:37138)
0620 15:05:04.954351 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.954468 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260620 15:05:04.955808 22099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:04.955359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33503 (local address 127.0.0.1:37146)
0620 15:05:04.955596 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:04.955680 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.1:33503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260620 15:05:04.986326 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 21959
I20260620 15:05:05.003882 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 21598
I20260620 15:05:05.005012 21660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:05.114205 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 21598
2026-06-20T15:05:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4252 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T15:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:05Z Disabled control of system clock
I20260620 15:05:05.191418 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35305
--webserver_interface=127.2.70.62
--webserver_port=0
--builtin_ntp_servers=127.2.70.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:35305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:05:05.517611 22141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:05.517907 22141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:05.518045 22141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:05.525254 22141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:05.525354 22141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:05.525430 22141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:05.525537 22141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:05.525609 22141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:05.534797 22141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:35305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:35305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:05.536056 22141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:05.537910 22141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:05.549547 22146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:05.549510 22147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:05.549573 22149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:05.550724 22141 server_base.cc:1034] running on GCE node
I20260620 15:05:05.551466 22141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:05.552867 22141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:05.554093 22141 hybrid_clock.cc:648] HybridClock initialized: now 1781967905554045 us; error 45 us; skew 500 ppm
I20260620 15:05:05.554461 22141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:05:05.556743 22141 webserver.cc:460] Webserver started at http://127.2.70.62:39971/ using document root <none> and password file <none>
I20260620 15:05:05.557425 22141 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:05.557585 22141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:05.557900 22141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:05.560670 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "28d68039e21f4124a806b7051fc392f7"
format_stamp: "Formatted at 2026-06-20 15:05:05 on dist-test-slave-n5vv"
server_key: "b15fb946da1e1372f81badb97db66607"
server_key_iv: "5f6151dde438874f1c399974d9c73601"
server_key_version: "encryptionkey@0"
I20260620 15:05:05.561470 22141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "28d68039e21f4124a806b7051fc392f7"
format_stamp: "Formatted at 2026-06-20 15:05:05 on dist-test-slave-n5vv"
server_key: "b15fb946da1e1372f81badb97db66607"
server_key_iv: "5f6151dde438874f1c399974d9c73601"
server_key_version: "encryptionkey@0"
I20260620 15:05:05.566548 22141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 15:05:05.569998 22155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:05.571412 22141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260620 15:05:05.571594 22141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "28d68039e21f4124a806b7051fc392f7"
format_stamp: "Formatted at 2026-06-20 15:05:05 on dist-test-slave-n5vv"
server_key: "b15fb946da1e1372f81badb97db66607"
server_key_iv: "5f6151dde438874f1c399974d9c73601"
server_key_version: "encryptionkey@0"
I20260620 15:05:05.571808 22141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:05.586467 22141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:05.587301 22141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:05.587597 22141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:05.606405 22141 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:35305
I20260620 15:05:05.606424 22206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:35305 every 8 connection(s)
I20260620 15:05:05.607774 22141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:05.611606 22207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:05.614856 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 22141
I20260620 15:05:05.615025 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:05.615464 2328 external_mini_cluster.cc:1428] Setting key 9b75936cf0343958d2318793579c4c2d
I20260620 15:05:05.625303 22207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7: Bootstrap starting.
I20260620 15:05:05.629338 22207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:05.630684 22207 log.cc:824] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:05.630988 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:05.617329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.2.70.62:35305)
0620 15:05:05.618446 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:05.618461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:05.618515 (+ 54us) server_negotiation.cc:407] Connection header received
0620 15:05:05.620679 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:05.620714 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:05.621392 (+ 678us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:05.621878 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:05.622484 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:05.624481 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:05.625876 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:05.626888 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:05.627114 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:05.628100 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:05.628110 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:05.628131 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:05.628848 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:05.628970 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:05.629003 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:05.629123 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:05.629322 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 15:05:05.629690 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":316,"threads_started":1}
I20260620 15:05:05.633919 22207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7: No bootstrap required, opened a new log
I20260620 15:05:05.641450 22207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d68039e21f4124a806b7051fc392f7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35305 } }
I20260620 15:05:05.641868 22207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:05.641942 22207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d68039e21f4124a806b7051fc392f7, State: Initialized, Role: FOLLOWER
I20260620 15:05:05.642511 22207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d68039e21f4124a806b7051fc392f7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35305 } }
I20260620 15:05:05.642716 22207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:05.642822 22207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:05.642939 22207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:05.645073 22207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d68039e21f4124a806b7051fc392f7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35305 } }
I20260620 15:05:05.645556 22207 leader_election.cc:302] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28d68039e21f4124a806b7051fc392f7; no voters:
I20260620 15:05:05.646234 22207 leader_election.cc:288] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:05.646356 22212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:05.647130 22212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [term 1 LEADER]: Becoming Leader. State: Replica: 28d68039e21f4124a806b7051fc392f7, State: Running, Role: LEADER
I20260620 15:05:05.647719 22212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d68039e21f4124a806b7051fc392f7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35305 } }
I20260620 15:05:05.648504 22207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:05.650877 22214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28d68039e21f4124a806b7051fc392f7. Latest consensus state: current_term: 1 leader_uuid: "28d68039e21f4124a806b7051fc392f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d68039e21f4124a806b7051fc392f7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35305 } } }
I20260620 15:05:05.650748 22213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28d68039e21f4124a806b7051fc392f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d68039e21f4124a806b7051fc392f7" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 35305 } } }
I20260620 15:05:05.651278 22214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:05.651346 22213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:05.653476 22218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:05.659520 22218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:05.668396 22218 catalog_manager.cc:1347] Generated new cluster ID: 046901f7f1be4a7798d2a9d0c64e8d6b
I20260620 15:05:05.668529 22218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:05.686071 22218 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:05.687448 22218 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:05.693316 22218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7: Generated new TSK 0
I20260620 15:05:05.694219 22218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:05.700966 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.1:0
--local_ip_for_outbound_sockets=127.2.70.1
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35305
--builtin_ntp_servers=127.2.70.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 15:05:06.033653 22231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:06.033947 22231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:06.034096 22231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:06.041486 22231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:06.041608 22231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:06.041785 22231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.1
I20260620 15:05:06.051306 22231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.70.1
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.1
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:06.052614 22231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:06.054761 22231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:06.068524 22236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:06.068538 22237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:06.068447 22239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:06.069454 22231 server_base.cc:1034] running on GCE node
I20260620 15:05:06.070070 22231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:06.070967 22231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:06.072229 22231 hybrid_clock.cc:648] HybridClock initialized: now 1781967906072146 us; error 29 us; skew 500 ppm
I20260620 15:05:06.072573 22231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:05:06.074836 22231 webserver.cc:460] Webserver started at http://127.2.70.1:37465/ using document root <none> and password file <none>
I20260620 15:05:06.075608 22231 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:06.075724 22231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:06.076056 22231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:06.079205 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/data/instance:
uuid: "d85d216997f94717ab5b8dc083199e43"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "56bf1e58d2ecfb6ce7eaf48824652328"
server_key_iv: "5b94a777a95c017c77d074ea08ec1cc6"
server_key_version: "encryptionkey@0"
I20260620 15:05:06.080024 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance:
uuid: "d85d216997f94717ab5b8dc083199e43"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "56bf1e58d2ecfb6ce7eaf48824652328"
server_key_iv: "5b94a777a95c017c77d074ea08ec1cc6"
server_key_version: "encryptionkey@0"
I20260620 15:05:06.085187 22231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260620 15:05:06.088462 22245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:06.089772 22231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:05:06.089946 22231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/wal
uuid: "d85d216997f94717ab5b8dc083199e43"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "56bf1e58d2ecfb6ce7eaf48824652328"
server_key_iv: "5b94a777a95c017c77d074ea08ec1cc6"
server_key_version: "encryptionkey@0"
I20260620 15:05:06.090157 22231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:06.117604 22231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:06.118592 22231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:06.118870 22231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:06.119860 22231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:05:06.121407 22231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:05:06.121538 22231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:06.121680 22231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:05:06.121783 22231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:06.155275 22231 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.1:33021
I20260620 15:05:06.155316 22357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.1:33021 every 8 connection(s)
I20260620 15:05:06.156665 22231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/data/info.pb
I20260620 15:05:06.158175 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 22231
I20260620 15:05:06.158336 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/wal/instance
I20260620 15:05:06.158878 2328 external_mini_cluster.cc:1428] Setting key 7c953472f8c6d146cdc0dea20e4f0902
I20260620 15:05:06.162352 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.2:0
--local_ip_for_outbound_sockets=127.2.70.2
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35305
--builtin_ntp_servers=127.2.70.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:05:06.172923 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:06.159315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:40907 (local address 127.2.70.62:35305)
0620 15:05:06.165893 (+ 6578us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:06.165901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:06.165923 (+ 22us) server_negotiation.cc:407] Connection header received
0620 15:05:06.165986 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:06.165993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:06.166078 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:06.166200 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:06.167298 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:06.168457 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:06.170123 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:06.170533 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:06.170675 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:06.172069 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:06.172075 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:06.172079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:06.172098 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:06.172145 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:06.172157 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:06.172255 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:06.172477 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 15:05:06.172578 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6435,"thread_start_us":6327,"threads_started":1}
I20260620 15:05:06.174006 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:06.159984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35305 (local address 127.2.70.1:40907)
0620 15:05:06.162042 (+ 2058us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:06.162113 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:06.165300 (+ 3187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:06.166414 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:06.166424 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:06.166524 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:06.167108 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:06.167125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:06.168613 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:06.168617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:06.169969 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:06.169981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:06.170774 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:06.170781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:06.171023 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:06.171737 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:05:06.171739 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:06.171784 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:05:06.171862 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:06.172255 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:06.172269 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:06.172279 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 15:05:06.172474 (+ 195us) client_negotiation.cc:240] Negotiation successful
0620 15:05:06.172885 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"thread_start_us":1005,"threads_started":1}
I20260620 15:05:06.175640 22358 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35305
I20260620 15:05:06.175956 22358 heartbeater.cc:464] Registering TS with master...
I20260620 15:05:06.176817 22358 heartbeater.cc:511] Master 127.2.70.62:35305 requested a full tablet report, sending...
I20260620 15:05:06.179091 22172 ts_manager.cc:194] Registered new tserver with Master: d85d216997f94717ab5b8dc083199e43 (127.2.70.1:33021)
I20260620 15:05:06.181362 22172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.1:40907
W20260620 15:05:06.496968 22362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:06.497248 22362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:06.497360 22362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:06.504554 22362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:06.504650 22362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:06.504753 22362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.2
I20260620 15:05:06.514931 22362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.70.2
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.2
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:06.516122 22362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:06.518003 22362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:06.530859 22367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:06.530943 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:06.530977 22368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:06.532317 22362 server_base.cc:1034] running on GCE node
I20260620 15:05:06.532884 22362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:06.533850 22362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:06.535054 22362 hybrid_clock.cc:648] HybridClock initialized: now 1781967906535004 us; error 45 us; skew 500 ppm
I20260620 15:05:06.535378 22362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:05:06.537714 22362 webserver.cc:460] Webserver started at http://127.2.70.2:41273/ using document root <none> and password file <none>
I20260620 15:05:06.538486 22362 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:06.538596 22362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:06.538930 22362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:06.541918 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/data/instance:
uuid: "2a84295ef7084530b2788a60c6f3aad3"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "0f491c01e7e2a03bc938e416d673541a"
server_key_iv: "75e2281ea0dda87e00742dd6ee8666eb"
server_key_version: "encryptionkey@0"
I20260620 15:05:06.542722 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance:
uuid: "2a84295ef7084530b2788a60c6f3aad3"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "0f491c01e7e2a03bc938e416d673541a"
server_key_iv: "75e2281ea0dda87e00742dd6ee8666eb"
server_key_version: "encryptionkey@0"
I20260620 15:05:06.547791 22362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:05:06.551271 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:06.552662 22362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 15:05:06.552863 22362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/wal
uuid: "2a84295ef7084530b2788a60c6f3aad3"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "0f491c01e7e2a03bc938e416d673541a"
server_key_iv: "75e2281ea0dda87e00742dd6ee8666eb"
server_key_version: "encryptionkey@0"
I20260620 15:05:06.553076 22362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:06.574957 22362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:06.575845 22362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:06.576119 22362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:06.577160 22362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:05:06.578919 22362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:05:06.579025 22362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:06.579149 22362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:05:06.579239 22362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:06.612947 22362 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.2:36465
I20260620 15:05:06.613013 22488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.2:36465 every 8 connection(s)
I20260620 15:05:06.614358 22362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/data/info.pb
I20260620 15:05:06.619634 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 22362
I20260620 15:05:06.619829 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/wal/instance
I20260620 15:05:06.620360 2328 external_mini_cluster.cc:1428] Setting key 2563362bcdc88a11e312ce3cfc597e30
I20260620 15:05:06.623246 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.70.3:0
--local_ip_for_outbound_sockets=127.2.70.3
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35305
--builtin_ntp_servers=127.2.70.20:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 15:05:06.628844 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:06.616975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.2:36337 (local address 127.2.70.62:35305)
0620 15:05:06.617184 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:06.617190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:06.618819 (+ 1629us) server_negotiation.cc:407] Connection header received
0620 15:05:06.621398 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:06.621403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:06.621478 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:06.621607 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:06.622930 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:06.624090 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:06.625778 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:06.626460 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:06.626593 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:06.627913 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:06.627922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:06.627930 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:06.627956 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:06.627999 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:06.628018 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:06.628107 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:06.628308 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 15:05:06.628422 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":67}
I20260620 15:05:06.629736 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:06.617562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35305 (local address 127.2.70.2:36337)
0620 15:05:06.618650 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:06.618707 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:06.621052 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:06.621874 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:06.621885 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:06.621999 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:06.622741 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:06.622762 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:06.624261 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:06.624276 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:06.625642 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:06.625654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:06.626675 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:06.626680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:06.626893 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:06.627602 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:05:06.627605 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:06.627644 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:05:06.627726 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:06.628119 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:06.628126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:06.628131 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:05:06.628290 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 15:05:06.628586 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":305,"threads_started":1}
I20260620 15:05:06.631266 22489 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35305
I20260620 15:05:06.631574 22489 heartbeater.cc:464] Registering TS with master...
I20260620 15:05:06.632302 22489 heartbeater.cc:511] Master 127.2.70.62:35305 requested a full tablet report, sending...
I20260620 15:05:06.633785 22172 ts_manager.cc:194] Registered new tserver with Master: 2a84295ef7084530b2788a60c6f3aad3 (127.2.70.2:36465)
I20260620 15:05:06.634999 22172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.2:36337
W20260620 15:05:06.950655 22492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:06.950923 22492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:06.951009 22492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:06.958443 22492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:06.958545 22492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:06.958652 22492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.70.3
I20260620 15:05:06.967643 22492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.70.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.70.3
--webserver_port=0
--tserver_master_addrs=127.2.70.62:35305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.70.3
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:06.968991 22492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:06.970722 22492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:06.985962 22497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:06.985988 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:06.986611 22500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:06.986734 22492 server_base.cc:1034] running on GCE node
I20260620 15:05:06.987382 22492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:06.988265 22492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:06.989557 22492 hybrid_clock.cc:648] HybridClock initialized: now 1781967906989497 us; error 44 us; skew 500 ppm
I20260620 15:05:06.989934 22492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260620 15:05:06.993117 22492 webserver.cc:460] Webserver started at http://127.2.70.3:46531/ using document root <none> and password file <none>
I20260620 15:05:06.993947 22492 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:06.994069 22492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:06.994390 22492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:06.997246 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/data/instance:
uuid: "71bc4cb454e8434184dc6a8fbb965347"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "d117043182e3a09de0570ba324a79c3e"
server_key_iv: "009d72661272c2532faa9d3cd531ba1c"
server_key_version: "encryptionkey@0"
I20260620 15:05:06.998054 22492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance:
uuid: "71bc4cb454e8434184dc6a8fbb965347"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "d117043182e3a09de0570ba324a79c3e"
server_key_iv: "009d72661272c2532faa9d3cd531ba1c"
server_key_version: "encryptionkey@0"
I20260620 15:05:07.002985 22492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:05:07.006667 22506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:07.007969 22492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:07.008131 22492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/wal
uuid: "71bc4cb454e8434184dc6a8fbb965347"
format_stamp: "Formatted at 2026-06-20 15:05:06 on dist-test-slave-n5vv"
server_key: "d117043182e3a09de0570ba324a79c3e"
server_key_iv: "009d72661272c2532faa9d3cd531ba1c"
server_key_version: "encryptionkey@0"
I20260620 15:05:07.008330 22492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:07.037118 22492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:07.038038 22492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:07.038308 22492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:07.039245 22492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 15:05:07.040715 22492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 15:05:07.040800 22492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:07.040951 22492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 15:05:07.041036 22492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:07.078061 22492 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.3:46087
I20260620 15:05:07.078147 22618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.3:46087 every 8 connection(s)
I20260620 15:05:07.079341 22492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/data/info.pb
I20260620 15:05:07.080507 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 22492
I20260620 15:05:07.080667 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/wal/instance
I20260620 15:05:07.081216 2328 external_mini_cluster.cc:1428] Setting key fb3d2e1ba8c98ab7ca7d21890e8db614
I20260620 15:05:07.091744 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.082178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.3:57257 (local address 127.2.70.62:35305)
0620 15:05:07.082357 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.082363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.084118 (+ 1755us) server_negotiation.cc:407] Connection header received
0620 15:05:07.085942 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.085946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.086009 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.086086 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:07.087089 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.087818 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.089323 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.089809 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.089902 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.091157 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:07.091161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:07.091164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:07.091179 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:07.091206 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:07.091216 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:07.091278 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.091446 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.091517 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:05:07.092694 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.082808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:35305 (local address 127.2.70.3:57257)
0620 15:05:07.083972 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:07.084021 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:07.085697 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:07.086281 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:07.086289 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:07.086400 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:07.086931 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.086950 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.087931 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.087935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.089214 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.089225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.089987 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.089992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.090187 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.090888 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 15:05:07.090891 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:07.090933 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 15:05:07.091004 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:07.091275 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:07.091281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:07.091286 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 15:05:07.091423 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 15:05:07.091685 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":352,"threads_started":1}
I20260620 15:05:07.094030 22619 heartbeater.cc:347] Connected to a master server at 127.2.70.62:35305
I20260620 15:05:07.094326 22619 heartbeater.cc:464] Registering TS with master...
I20260620 15:05:07.095118 22619 heartbeater.cc:511] Master 127.2.70.62:35305 requested a full tablet report, sending...
I20260620 15:05:07.096531 22172 ts_manager.cc:194] Registered new tserver with Master: 71bc4cb454e8434184dc6a8fbb965347 (127.2.70.3:46087)
I20260620 15:05:07.097224 22172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.70.3:57257
I20260620 15:05:07.108397 2328 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 15:05:07.117764 22360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.112156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.2.70.62:35305)
0620 15:05:07.112340 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.112344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.112863 (+ 519us) server_negotiation.cc:407] Connection header received
0620 15:05:07.113011 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.113015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.113085 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.113163 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:07.113637 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.114314 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.115587 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.115966 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.116109 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.117074 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:07.117079 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:07.117082 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:07.117099 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:07.117127 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:07.117139 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:07.117224 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.117366 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.117461 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:05:07.121281 22172 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 15:05:07.127933 22172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 15:05:07.152629 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.138980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.1:33021 (local address 127.0.0.1:59486)
0620 15:05:07.140091 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:07.140138 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:07.140336 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:07.143150 (+ 2814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:07.143157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:07.143188 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:07.143711 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.143725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.146652 (+ 2927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.146661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.149768 (+ 3107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.149784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.151643 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.151649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.151849 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.151895 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 15:05:07.152016 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 15:05:07.152232 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":3482,"threads_started":1}
I20260620 15:05:07.153620 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.139220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.2.70.1:33021)
0620 15:05:07.142549 (+ 3329us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.142556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.142582 (+ 26us) server_negotiation.cc:407] Connection header received
0620 15:05:07.142664 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.142669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.142893 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.144001 (+ 1108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:07.144319 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.146453 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.150015 (+ 3562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.151445 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.151589 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.151841 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.153040 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.153247 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3185,"thread_start_us":3083,"threads_started":1}
I20260620 15:05:07.154018 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.139928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.2.70.2:36465)
0620 15:05:07.142268 (+ 2340us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.142281 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.142855 (+ 574us) server_negotiation.cc:407] Connection header received
0620 15:05:07.143257 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.143267 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.143556 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.143771 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:07.144972 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.147163 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.151076 (+ 3913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.152481 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.152606 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.152839 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.153507 (+ 668us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.153680 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2112,"thread_start_us":1956,"threads_started":1}
I20260620 15:05:07.153945 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.139257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:36465 (local address 127.0.0.1:43880)
0620 15:05:07.142750 (+ 3493us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:07.142770 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:07.142957 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:07.144123 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:07.144131 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:07.144245 (+ 114us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:07.144733 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.144749 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.147849 (+ 3100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.147857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.150873 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.150889 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.152982 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.152988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.153299 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.153323 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 15:05:07.153426 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 15:05:07.153568 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3341,"mutex_wait_us":114,"thread_start_us":903,"threads_started":1}
I20260620 15:05:07.157248 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.142895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:46087 (local address 127.0.0.1:46362)
0620 15:05:07.149182 (+ 6287us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:07.149201 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:07.149341 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:07.150089 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:07.150096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:07.150123 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:07.150542 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.150555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.152219 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.152224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.154413 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.154429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.155987 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.155993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.156184 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.156225 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:05:07.156313 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 15:05:07.156442 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6149,"mutex_wait_us":55,"thread_start_us":6006,"threads_started":1}
I20260620 15:05:07.157944 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.143185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.2.70.3:46087)
0620 15:05:07.148906 (+ 5721us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.148916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.149471 (+ 555us) server_negotiation.cc:407] Connection header received
0620 15:05:07.149608 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.149615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.149912 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.150117 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:07.150743 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.152064 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.154598 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.155793 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.155926 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.156168 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.156726 (+ 558us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.156899 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5546,"thread_start_us":5336,"threads_started":1}
I20260620 15:05:07.158684 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 55d13bb25895402ba55cc9aa4e157604 (DEFAULT_TABLE table=test-table [id=14ce770e809e4ab6ac55f68c50014128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:07.160247 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d13bb25895402ba55cc9aa4e157604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:07.161806 22554 tablet_service.cc:1467] Processing CreateTablet for tablet 55d13bb25895402ba55cc9aa4e157604 (DEFAULT_TABLE table=test-table [id=14ce770e809e4ab6ac55f68c50014128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:07.161886 22293 tablet_service.cc:1467] Processing CreateTablet for tablet 55d13bb25895402ba55cc9aa4e157604 (DEFAULT_TABLE table=test-table [id=14ce770e809e4ab6ac55f68c50014128]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 15:05:07.162943 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d13bb25895402ba55cc9aa4e157604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:07.163195 22293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55d13bb25895402ba55cc9aa4e157604. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:07.173946 22637 tablet_bootstrap.cc:492] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Bootstrap starting.
I20260620 15:05:07.178367 22637 tablet_bootstrap.cc:654] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:07.179244 22639 tablet_bootstrap.cc:492] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: Bootstrap starting.
I20260620 15:05:07.179692 22637 log.cc:824] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:07.183120 22639 tablet_bootstrap.cc:654] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:07.183226 22637 tablet_bootstrap.cc:492] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: No bootstrap required, opened a new log
I20260620 15:05:07.183413 22638 tablet_bootstrap.cc:492] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: Bootstrap starting.
I20260620 15:05:07.183481 22637 ts_tablet_manager.cc:1397] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260620 15:05:07.184590 22358 heartbeater.cc:503] Master 127.2.70.62:35305 was elected leader, sending a full tablet report...
I20260620 15:05:07.184835 22639 log.cc:824] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:07.187230 22638 tablet_bootstrap.cc:654] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:07.188515 22638 log.cc:824] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:07.192191 22639 tablet_bootstrap.cc:492] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: No bootstrap required, opened a new log
I20260620 15:05:07.192533 22639 ts_tablet_manager.cc:1397] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260620 15:05:07.195489 22637 raft_consensus.cc:348] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.195955 22637 raft_consensus.cc:374] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:07.196123 22638 tablet_bootstrap.cc:492] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: No bootstrap required, opened a new log
I20260620 15:05:07.196094 22637 raft_consensus.cc:729] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71bc4cb454e8434184dc6a8fbb965347, State: Initialized, Role: FOLLOWER
I20260620 15:05:07.196420 22638 ts_tablet_manager.cc:1397] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20260620 15:05:07.196765 22637 consensus_queue.cc:260] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.198231 22619 heartbeater.cc:503] Master 127.2.70.62:35305 was elected leader, sending a full tablet report...
I20260620 15:05:07.198952 22637 ts_tablet_manager.cc:1428] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260620 15:05:07.202957 22639 raft_consensus.cc:348] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.203581 22639 raft_consensus.cc:374] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:07.203742 22639 raft_consensus.cc:729] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d85d216997f94717ab5b8dc083199e43, State: Initialized, Role: FOLLOWER
I20260620 15:05:07.204407 22639 consensus_queue.cc:260] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.205547 22638 raft_consensus.cc:348] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.205950 22638 raft_consensus.cc:374] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:07.206065 22638 raft_consensus.cc:729] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a84295ef7084530b2788a60c6f3aad3, State: Initialized, Role: FOLLOWER
I20260620 15:05:07.206223 22639 ts_tablet_manager.cc:1428] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260620 15:05:07.206717 22638 consensus_queue.cc:260] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.207851 22489 heartbeater.cc:503] Master 127.2.70.62:35305 was elected leader, sending a full tablet report...
I20260620 15:05:07.208487 22638 ts_tablet_manager.cc:1428] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: Time spent starting tablet: real 0.012s user 0.000s sys 0.011s
I20260620 15:05:07.270805 22644 raft_consensus.cc:482] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 15:05:07.271136 22644 raft_consensus.cc:504] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.272881 22644 leader_election.cc:288] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a84295ef7084530b2788a60c6f3aad3 (127.2.70.2:36465), 71bc4cb454e8434184dc6a8fbb965347 (127.2.70.3:46087)
I20260620 15:05:07.280442 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.273310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:54357 (local address 127.2.70.2:36465)
0620 15:05:07.273587 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.273595 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.274333 (+ 738us) server_negotiation.cc:407] Connection header received
0620 15:05:07.274628 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.274635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.274732 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.274808 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:07.275896 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.276801 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.278795 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.279546 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.279672 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.279765 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.280081 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.280194 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260620 15:05:07.280568 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.273368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.70.1:55419 (local address 127.2.70.3:46087)
0620 15:05:07.273601 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.273609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.274501 (+ 892us) server_negotiation.cc:407] Connection header received
0620 15:05:07.274673 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.274679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.274782 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.274886 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 15:05:07.275841 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.276819 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.278895 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.279661 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.279824 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.279938 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.280222 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.280342 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:05:07.280644 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.273315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.2:36465 (local address 127.2.70.1:54357)
0620 15:05:07.274225 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:07.274250 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:07.274424 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:07.275050 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:07.275055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:07.275085 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:07.275642 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.275649 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.276935 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.276941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.278610 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.278625 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.279723 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.279728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.279902 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.279943 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 15:05:07.280042 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 15:05:07.280179 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"mutex_wait_us":5,"spinlock_wait_cycles":32512,"thread_start_us":541,"threads_started":1}
I20260620 15:05:07.280684 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.273381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.3:46087 (local address 127.2.70.1:55419)
0620 15:05:07.274369 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:07.274390 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:07.274527 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:07.274988 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:07.274996 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:07.275058 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 15:05:07.275636 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.275649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.276978 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.276983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.278687 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:07.278700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.279850 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.279856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:07.280074 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.280097 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 15:05:07.280193 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 15:05:07.280319 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":66,"thread_start_us":504,"threads_started":1}
I20260620 15:05:07.281729 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d13bb25895402ba55cc9aa4e157604" candidate_uuid: "d85d216997f94717ab5b8dc083199e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a84295ef7084530b2788a60c6f3aad3" is_pre_election: true
I20260620 15:05:07.281762 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d13bb25895402ba55cc9aa4e157604" candidate_uuid: "d85d216997f94717ab5b8dc083199e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71bc4cb454e8434184dc6a8fbb965347" is_pre_election: true
I20260620 15:05:07.282224 22574 raft_consensus.cc:2454] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85d216997f94717ab5b8dc083199e43 in term 0.
I20260620 15:05:07.282232 22444 raft_consensus.cc:2454] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d85d216997f94717ab5b8dc083199e43 in term 0.
I20260620 15:05:07.282876 22246 leader_election.cc:302] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a84295ef7084530b2788a60c6f3aad3, 71bc4cb454e8434184dc6a8fbb965347, d85d216997f94717ab5b8dc083199e43; no voters:
I20260620 15:05:07.283242 22644 raft_consensus.cc:2781] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 15:05:07.283397 22644 raft_consensus.cc:482] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 15:05:07.283499 22644 raft_consensus.cc:3037] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:07.285723 22644 raft_consensus.cc:504] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.286341 22644 leader_election.cc:288] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [CANDIDATE]: Term 1 election: Requested vote from peers 2a84295ef7084530b2788a60c6f3aad3 (127.2.70.2:36465), 71bc4cb454e8434184dc6a8fbb965347 (127.2.70.3:46087)
I20260620 15:05:07.286744 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d13bb25895402ba55cc9aa4e157604" candidate_uuid: "d85d216997f94717ab5b8dc083199e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a84295ef7084530b2788a60c6f3aad3"
I20260620 15:05:07.286754 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55d13bb25895402ba55cc9aa4e157604" candidate_uuid: "d85d216997f94717ab5b8dc083199e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71bc4cb454e8434184dc6a8fbb965347"
I20260620 15:05:07.286926 22444 raft_consensus.cc:3037] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:07.286942 22574 raft_consensus.cc:3037] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:07.289063 22574 raft_consensus.cc:2454] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85d216997f94717ab5b8dc083199e43 in term 1.
I20260620 15:05:07.289070 22444 raft_consensus.cc:2454] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d85d216997f94717ab5b8dc083199e43 in term 1.
I20260620 15:05:07.289458 22246 leader_election.cc:302] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71bc4cb454e8434184dc6a8fbb965347, d85d216997f94717ab5b8dc083199e43; no voters:
I20260620 15:05:07.289793 22644 raft_consensus.cc:2781] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:07.290170 22644 raft_consensus.cc:686] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 1 LEADER]: Becoming Leader. State: Replica: d85d216997f94717ab5b8dc083199e43, State: Running, Role: LEADER
I20260620 15:05:07.290618 22644 consensus_queue.cc:237] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } }
I20260620 15:05:07.294831 22172 catalog_manager.cc:5495] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to d85d216997f94717ab5b8dc083199e43 (127.2.70.1). New cstate: current_term: 1 leader_uuid: "d85d216997f94717ab5b8dc083199e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d85d216997f94717ab5b8dc083199e43" member_type: VOTER last_known_addr { host: "127.2.70.1" port: 33021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 } health_report { overall_health: UNKNOWN } } }
I20260620 15:05:07.317975 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.312841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59488 (local address 127.2.70.1:33021)
0620 15:05:07.313024 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.313029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.313198 (+ 169us) server_negotiation.cc:407] Connection header received
0620 15:05:07.313359 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.313362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.313452 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.313628 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:07.313966 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.314660 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.315954 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.316355 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.316455 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.316864 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:07.317344 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:07.317475 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.317608 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.317670 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 15:05:07.325911 22574 raft_consensus.cc:1261] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 1 FOLLOWER]: Refusing update from remote peer d85d216997f94717ab5b8dc083199e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:07.325950 22444 raft_consensus.cc:1261] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 1 FOLLOWER]: Refusing update from remote peer d85d216997f94717ab5b8dc083199e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:07.326905 22648 consensus_queue.cc:1035] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a84295ef7084530b2788a60c6f3aad3" member_type: VOTER last_known_addr { host: "127.2.70.2" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:05:07.327263 22644 consensus_queue.cc:1035] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71bc4cb454e8434184dc6a8fbb965347" member_type: VOTER last_known_addr { host: "127.2.70.3" port: 46087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 15:05:07.330948 22620 tablet.cc:2369] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:07.342691 22652 mvcc.cc:205] Tried to move back new op lower bound from 7298940548397367296 to 7298940548265889792. Current Snapshot: MvccSnapshot[applied={T|T < 7298940548397367296}]
I20260620 15:05:07.353614 22171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:05:07.353911 22171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 15:05:07.357654 22171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28d68039e21f4124a806b7051fc392f7: Sending DeleteTablet for 3 replicas of tablet 55d13bb25895402ba55cc9aa4e157604
I20260620 15:05:07.358923 22293 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d13bb25895402ba55cc9aa4e157604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:07 UTC) from {username='slave'} at 127.0.0.1:59486
I20260620 15:05:07.359055 22424 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d13bb25895402ba55cc9aa4e157604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:07 UTC) from {username='slave'} at 127.0.0.1:43880
I20260620 15:05:07.359086 22554 tablet_service.cc:1514] Processing DeleteTablet for tablet 55d13bb25895402ba55cc9aa4e157604 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 15:05:07 UTC) from {username='slave'} at 127.0.0.1:46362
I20260620 15:05:07.359892 22661 tablet_replica.cc:330] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: stopping tablet replica
I20260620 15:05:07.360482 22661 raft_consensus.cc:2229] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 1 LEADER]: Raft consensus shutting down.
I20260620 15:05:07.360597 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22231
I20260620 15:05:07.361217 22661 raft_consensus.cc:2258] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:07.369374 22661 ts_tablet_manager.cc:1905] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:07.378275 22662 tablet_replica.cc:330] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: stopping tablet replica
I20260620 15:05:07.379315 22662 raft_consensus.cc:2229] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:05:07.379963 22662 raft_consensus.cc:2258] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:07.380048 22661 ts_tablet_manager.cc:1918] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:07.380272 22661 log.cc:1196] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-0/wal/wals/55d13bb25895402ba55cc9aa4e157604
I20260620 15:05:07.380939 22661 ts_tablet_manager.cc:1939] T 55d13bb25895402ba55cc9aa4e157604 P d85d216997f94717ab5b8dc083199e43: Deleting consensus metadata
I20260620 15:05:07.382357 22662 ts_tablet_manager.cc:1905] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:07.383296 22156 catalog_manager.cc:4841] TS d85d216997f94717ab5b8dc083199e43 (127.2.70.1:33021): tablet 55d13bb25895402ba55cc9aa4e157604 (table test-table [id=14ce770e809e4ab6ac55f68c50014128]) successfully deleted
I20260620 15:05:07.387807 22632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.361768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59490 (local address 127.2.70.1:33021)
0620 15:05:07.361974 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.361983 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.374154 (+ 12171us) server_negotiation.cc:407] Connection header received
0620 15:05:07.374519 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.374529 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.374673 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.374768 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:07.375769 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.377497 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.380136 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.380942 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.381048 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.386021 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:07.386035 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:07.386055 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:07.386842 (+ 787us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:07.386908 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:07.386939 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:07.387017 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.387316 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.387428 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 15:05:07.388453 22662 ts_tablet_manager.cc:1918] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:07.388672 22662 log.cc:1196] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-1/wal/wals/55d13bb25895402ba55cc9aa4e157604
I20260620 15:05:07.389242 22353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:07.389369 22662 ts_tablet_manager.cc:1939] T 55d13bb25895402ba55cc9aa4e157604 P 2a84295ef7084530b2788a60c6f3aad3: Deleting consensus metadata
I20260620 15:05:07.390169 22663 tablet_replica.cc:330] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: stopping tablet replica
I20260620 15:05:07.390566 22663 raft_consensus.cc:2229] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 15:05:07.391006 22663 raft_consensus.cc:2258] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 15:05:07.391719 22159 catalog_manager.cc:4841] TS 2a84295ef7084530b2788a60c6f3aad3 (127.2.70.2:36465): tablet 55d13bb25895402ba55cc9aa4e157604 (table test-table [id=14ce770e809e4ab6ac55f68c50014128]) successfully deleted
I20260620 15:05:07.393365 22663 ts_tablet_manager.cc:1905] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 15:05:07.400612 22663 ts_tablet_manager.cc:1918] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 15:05:07.400780 22663 log.cc:1196] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Deleting WAL directory at /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781967589245963-2328-0/minicluster-data/ts-2/wal/wals/55d13bb25895402ba55cc9aa4e157604
I20260620 15:05:07.401295 22663 ts_tablet_manager.cc:1939] T 55d13bb25895402ba55cc9aa4e157604 P 71bc4cb454e8434184dc6a8fbb965347: Deleting consensus metadata
I20260620 15:05:07.403205 22156 catalog_manager.cc:4841] TS 71bc4cb454e8434184dc6a8fbb965347 (127.2.70.3:46087): tablet 55d13bb25895402ba55cc9aa4e157604 (table test-table [id=14ce770e809e4ab6ac55f68c50014128]) successfully deleted
I20260620 15:05:07.490347 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 22231
I20260620 15:05:07.509325 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22362
I20260620 15:05:07.516036 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.510065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.2.70.2:36465)
0620 15:05:07.510232 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.510236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.510257 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:05:07.510447 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.510451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.510527 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.510623 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:07.511047 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.511758 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.513160 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.513620 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.513746 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.514727 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:07.514736 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:07.514750 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:07.515300 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:07.515345 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:07.515364 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:07.515500 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.515620 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.515723 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:05:07.516827 22484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:07.605695 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 22362
I20260620 15:05:07.625869 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22492
I20260620 15:05:07.637822 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:07.626608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.2.70.3:46087)
0620 15:05:07.626793 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:07.626799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:07.626816 (+ 17us) server_negotiation.cc:407] Connection header received
0620 15:05:07.626983 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:07.626987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:07.627069 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:07.627155 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:07.627492 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.628221 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.629457 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:07.629954 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:07.630062 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:07.636489 (+ 6427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:07.636499 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:07.636514 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:07.637090 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:07.637154 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:07.637175 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:07.637314 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:07.637425 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 15:05:07.637576 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:05:07.638495 22614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:07.726644 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 22492
I20260620 15:05:07.743216 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22141
I20260620 15:05:07.744334 22202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:07.853557 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 22141
2026-06-20T15:05:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2737 ms)
[----------] 6 tests from EncryptionPolicyTest (21304 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:07 dist-test-slave-n5vv krb5kdc[22671](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:07 dist-test-slave-n5vv krb5kdc[22671](info): set up 2 sockets
Jun 20 15:05:07 dist-test-slave-n5vv krb5kdc[22671](info): commencing operation
krb5kdc: starting...
W20260620 15:05:08.985972 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:09 dist-test-slave-n5vv krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:09.239395 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:09.574587 22684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:09.574885 22684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:09.575048 22684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:09.582401 22684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:09.582520 22684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:09.582621 22684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:09.582731 22684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:09.582803 22684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:09.592924 22684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:09.594307 22684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:09.595659 22684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:09.608083 22689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:09.608125 22690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:09.608170 22692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:09.609421 22684 server_base.cc:1034] running on GCE node
I20260620 15:05:09.610304 22684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:09.611835 22684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:09.613073 22684 hybrid_clock.cc:648] HybridClock initialized: now 1781967909613025 us; error 40 us; skew 500 ppm
I20260620 15:05:09.613466 22684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:09 dist-test-slave-n5vv krb5kdc[22671](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967909, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:09.616791 22684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:09.618418 22684 webserver.cc:460] Webserver started at http://127.0.0.1:43449/ using document root <none> and password file <none>
I20260620 15:05:09.619143 22684 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:09.619284 22684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:09.619594 22684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:09.622658 22684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "54f310d05022460cb52c81a82b0708bb"
format_stamp: "Formatted at 2026-06-20 15:05:09 on dist-test-slave-n5vv"
server_key: "3af96dafa19d279e000ea4b53622b590"
server_key_iv: "b01e2ebaa42e96f0c9b733d7459edbc3"
server_key_version: "encryptionkey@0"
I20260620 15:05:09.623449 22684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "54f310d05022460cb52c81a82b0708bb"
format_stamp: "Formatted at 2026-06-20 15:05:09 on dist-test-slave-n5vv"
server_key: "3af96dafa19d279e000ea4b53622b590"
server_key_iv: "b01e2ebaa42e96f0c9b733d7459edbc3"
server_key_version: "encryptionkey@0"
I20260620 15:05:09.628693 22684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260620 15:05:09.632323 22699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:09.633912 22684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:05:09.634061 22684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "54f310d05022460cb52c81a82b0708bb"
format_stamp: "Formatted at 2026-06-20 15:05:09 on dist-test-slave-n5vv"
server_key: "3af96dafa19d279e000ea4b53622b590"
server_key_iv: "b01e2ebaa42e96f0c9b733d7459edbc3"
server_key_version: "encryptionkey@0"
I20260620 15:05:09.634243 22684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:09.648746 22684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:09.653354 22684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:09.653671 22684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:09.672433 22684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42963
I20260620 15:05:09.672461 22750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42963 every 8 connection(s)
I20260620 15:05:09.674149 22684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:09.678110 22751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:09.684216 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 22684
I20260620 15:05:09.684451 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:09.685253 2328 external_mini_cluster.cc:1428] Setting key 10d347858bb70db42a248e9f1c089fba
I20260620 15:05:09.689579 22751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb: Bootstrap starting.
I20260620 15:05:09.693325 22751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:09.694645 22751 log.cc:824] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:09.697755 22751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb: No bootstrap required, opened a new log
Jun 20 15:05:09 dist-test-slave-n5vv krb5kdc[22671](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967909, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:05:09.705287 22751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f310d05022460cb52c81a82b0708bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260620 15:05:09.705727 22751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:09.705843 22751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f310d05022460cb52c81a82b0708bb, State: Initialized, Role: FOLLOWER
I20260620 15:05:09.706491 22751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f310d05022460cb52c81a82b0708bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260620 15:05:09.706748 22751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:09.706892 22751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:09.707119 22751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:09.708171 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:09.687322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60441 (local address 127.0.0.1:42963)
0620 15:05:09.688125 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:09.688136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:09.688176 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:05:09.689928 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:09.689950 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:09.690344 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:09.690667 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:09.692531 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:09.693985 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:09.695827 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:09.696502 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:09.696634 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:09.700744 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:09.700754 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:09.700768 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:09.700806 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:09.703903 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:09.704683 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:09.704690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:09.704696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:09.704856 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:09.705314 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:09.705318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:09.705320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:09.705997 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:09.706356 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:09.706780 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 15:05:09.707128 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":394,"threads_started":1}
I20260620 15:05:09.709654 22751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f310d05022460cb52c81a82b0708bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260620 15:05:09.710141 22751 leader_election.cc:302] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54f310d05022460cb52c81a82b0708bb; no voters:
I20260620 15:05:09.710783 22751 leader_election.cc:288] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:09.710904 22756 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:09.712242 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:09.712298 22756 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [term 1 LEADER]: Becoming Leader. State: Replica: 54f310d05022460cb52c81a82b0708bb, State: Running, Role: LEADER
I20260620 15:05:09.712855 22756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f310d05022460cb52c81a82b0708bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } }
I20260620 15:05:09.713125 22751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:09.720846 22757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54f310d05022460cb52c81a82b0708bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f310d05022460cb52c81a82b0708bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } } }
I20260620 15:05:09.721366 22757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:09.720875 22758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f310d05022460cb52c81a82b0708bb. Latest consensus state: current_term: 1 leader_uuid: "54f310d05022460cb52c81a82b0708bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f310d05022460cb52c81a82b0708bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42963 } } }
I20260620 15:05:09.721673 22758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:09.735778 22771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:09.741473 22771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:09.748917 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:09.721744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40643 (local address 127.0.0.1:42963)
0620 15:05:09.721937 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:09.721946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:09.732006 (+ 10060us) server_negotiation.cc:407] Connection header received
0620 15:05:09.732647 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:09.732654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:09.732769 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:09.732867 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:09.735568 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:09.736824 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:09.738494 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:09.739159 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:09.739314 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:09.742002 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:09.742007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:09.742011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:09.742055 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:09.745102 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:09.746066 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:09.746073 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:09.746076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:09.746249 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:09.746787 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:09.746793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:09.746797 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:09.747302 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:09.747608 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:09.748042 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 15:05:09.748493 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:05:09.751962 22771 catalog_manager.cc:1347] Generated new cluster ID: 517f10854c554c1280e7df6a493932eb
I20260620 15:05:09.752081 22771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:09.768786 22771 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:09.770674 22771 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:09.778927 22771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54f310d05022460cb52c81a82b0708bb: Generated new TSK 0
I20260620 15:05:09.780076 22771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:09.792444 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22684
I20260620 15:05:09.794503 22746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:09.885391 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 22684
W20260620 15:05:09.902635 22669 connection.cc:535] client connection to 127.0.0.1:42963 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42963): Connection reset by peer (error 104)
2026-06-20T15:05:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2031 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:09 dist-test-slave-n5vv krb5kdc[22783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:09 dist-test-slave-n5vv krb5kdc[22783](info): set up 2 sockets
Jun 20 15:05:09 dist-test-slave-n5vv krb5kdc[22783](info): commencing operation
krb5kdc: starting...
W20260620 15:05:11.025801 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:11 dist-test-slave-n5vv krb5kdc[22783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:11.287212 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:11.619074 22796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:11.619431 22796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:11.619589 22796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:11.626890 22796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:11.627040 22796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:11.627146 22796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:11.627247 22796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:11.627321 22796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:11.638079 22796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:11.639424 22796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:11.640776 22796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:11.653806 22801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:11.653825 22804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:11.653846 22802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:11.659024 22796 server_base.cc:1034] running on GCE node
I20260620 15:05:11.659931 22796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:11.661538 22796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:11.662797 22796 hybrid_clock.cc:648] HybridClock initialized: now 1781967911662751 us; error 46 us; skew 500 ppm
I20260620 15:05:11.663187 22796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:11 dist-test-slave-n5vv krb5kdc[22783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967911, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:11.666384 22796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:11.667883 22796 webserver.cc:460] Webserver started at http://127.0.0.1:38871/ using document root <none> and password file <none>
I20260620 15:05:11.668641 22796 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:11.668789 22796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:11.669096 22796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:11.672302 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "c39237383ae143fe8a607f3420bb38cd"
format_stamp: "Formatted at 2026-06-20 15:05:11 on dist-test-slave-n5vv"
server_key: "c8ace2dbebdd2bf0e3fddedf612fcb3a"
server_key_iv: "e563d90073c65530c24c6742b3df3a9d"
server_key_version: "encryptionkey@0"
I20260620 15:05:11.673082 22796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "c39237383ae143fe8a607f3420bb38cd"
format_stamp: "Formatted at 2026-06-20 15:05:11 on dist-test-slave-n5vv"
server_key: "c8ace2dbebdd2bf0e3fddedf612fcb3a"
server_key_iv: "e563d90073c65530c24c6742b3df3a9d"
server_key_version: "encryptionkey@0"
I20260620 15:05:11.678587 22796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:05:11.682478 22811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:11.683954 22796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:11.684141 22796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "c39237383ae143fe8a607f3420bb38cd"
format_stamp: "Formatted at 2026-06-20 15:05:11 on dist-test-slave-n5vv"
server_key: "c8ace2dbebdd2bf0e3fddedf612fcb3a"
server_key_iv: "e563d90073c65530c24c6742b3df3a9d"
server_key_version: "encryptionkey@0"
I20260620 15:05:11.684382 22796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:11.697414 22796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:11.701897 22796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:11.702185 22796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:11.720600 22796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37305
I20260620 15:05:11.720643 22862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37305 every 8 connection(s)
I20260620 15:05:11.722029 22796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:11.726079 22863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:11.731223 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 22796
I20260620 15:05:11.731415 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:11.731842 2328 external_mini_cluster.cc:1428] Setting key e286c8f1c1f701dac9d7f4f54b05e110
I20260620 15:05:11.738130 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd: Bootstrap starting.
I20260620 15:05:11.742141 22863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:11.743630 22863 log.cc:824] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:11.745998 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:11.733630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44723 (local address 127.0.0.1:37305)
0620 15:05:11.734460 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:11.734471 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:11.734516 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:05:11.736358 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:11.736380 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:11.736759 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:11.737060 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:11.737717 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:11.739099 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:11.741199 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:11.741913 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:11.742044 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:11.743456 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:11.743463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:11.743476 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:11.744059 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:11.744182 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:11.744232 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:11.744438 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:11.744711 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 15:05:11.744916 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":376,"threads_started":1}
I20260620 15:05:11.747117 22863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd: No bootstrap required, opened a new log
I20260620 15:05:11.749930 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:11.756667 22863 raft_consensus.cc:348] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39237383ae143fe8a607f3420bb38cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 15:05:11.757140 22863 raft_consensus.cc:374] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:11.757272 22863 raft_consensus.cc:729] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39237383ae143fe8a607f3420bb38cd, State: Initialized, Role: FOLLOWER
I20260620 15:05:11.758049 22863 consensus_queue.cc:260] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39237383ae143fe8a607f3420bb38cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 15:05:11.758322 22863 raft_consensus.cc:388] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:11.758491 22863 raft_consensus.cc:482] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:11.758711 22863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:11.760962 22866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:11.754546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40477 (local address 127.0.0.1:37305)
0620 15:05:11.754723 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:11.754731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:11.755197 (+ 466us) server_negotiation.cc:407] Connection header received
0620 15:05:11.755422 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:11.755426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:11.755491 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:11.755595 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:11.756108 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:11.756858 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:11.758429 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:11.758956 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:11.759080 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:11.760241 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:11.760246 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:11.760250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:11.760279 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:11.760317 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:11.760332 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:11.760454 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:11.760631 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 15:05:11.760682 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:05:11.761519 22863 raft_consensus.cc:504] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39237383ae143fe8a607f3420bb38cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 15:05:11.762107 22863 leader_election.cc:302] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c39237383ae143fe8a607f3420bb38cd; no voters:
I20260620 15:05:11.762854 22863 leader_election.cc:288] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:11.762965 22873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:11.763707 22873 raft_consensus.cc:686] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [term 1 LEADER]: Becoming Leader. State: Replica: c39237383ae143fe8a607f3420bb38cd, State: Running, Role: LEADER
I20260620 15:05:11.764338 22873 consensus_queue.cc:237] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39237383ae143fe8a607f3420bb38cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } }
I20260620 15:05:11.765385 22863 sys_catalog.cc:564] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:11.767647 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader c39237383ae143fe8a607f3420bb38cd. Latest consensus state: current_term: 1 leader_uuid: "c39237383ae143fe8a607f3420bb38cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39237383ae143fe8a607f3420bb38cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260620 15:05:11.768214 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:11.769742 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c39237383ae143fe8a607f3420bb38cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39237383ae143fe8a607f3420bb38cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37305 } } }
I20260620 15:05:11.770006 22881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:11.770648 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:11.775789 22881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:11.785449 22881 catalog_manager.cc:1347] Generated new cluster ID: a4d7e43a8ce647dabbcdddfe879f9783
I20260620 15:05:11.785604 22881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:11.822503 22881 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:11.823902 22881 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:11.834311 22881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c39237383ae143fe8a607f3420bb38cd: Generated new TSK 0
I20260620 15:05:11.835182 22881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:11.846174 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22796
W20260620 15:05:11.847347 22858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44723
W20260620 15:05:11.848167 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:11.848361 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 22796
W20260620 15:05:11.865223 22781 connection.cc:535] client connection to 127.0.0.1:37305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37305): Connection reset by peer (error 104)
2026-06-20T15:05:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:11 dist-test-slave-n5vv krb5kdc[22894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:11 dist-test-slave-n5vv krb5kdc[22894](info): set up 2 sockets
Jun 20 15:05:11 dist-test-slave-n5vv krb5kdc[22894](info): commencing operation
krb5kdc: starting...
W20260620 15:05:12.961951 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:13 dist-test-slave-n5vv krb5kdc[22894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:13.212348 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:13.547933 22907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:13.548352 22907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:13.548523 22907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:13.557111 22907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:13.557302 22907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:13.557415 22907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:13.557552 22907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:13.557657 22907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:13.569036 22907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:13.570485 22907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:13.571844 22907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:13.584527 22915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:13.584473 22912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:13.584502 22913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:13.585103 22907 server_base.cc:1034] running on GCE node
I20260620 15:05:13.586104 22907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:13.587543 22907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:13.588780 22907 hybrid_clock.cc:648] HybridClock initialized: now 1781967913588707 us; error 55 us; skew 500 ppm
I20260620 15:05:13.589284 22907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:13 dist-test-slave-n5vv krb5kdc[22894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:13.599025 22907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:13.600630 22907 webserver.cc:460] Webserver started at http://127.0.0.1:45165/ using document root <none> and password file <none>
I20260620 15:05:13.601367 22907 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:13.601541 22907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:13.601840 22907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:13.604913 22907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da"
format_stamp: "Formatted at 2026-06-20 15:05:13 on dist-test-slave-n5vv"
server_key: "74de18f2b12b7eef43f87cf6ecdc0dfd"
server_key_iv: "f11c34c9fc9d68a7b2ee62f9e261f522"
server_key_version: "encryptionkey@0"
I20260620 15:05:13.605732 22907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da"
format_stamp: "Formatted at 2026-06-20 15:05:13 on dist-test-slave-n5vv"
server_key: "74de18f2b12b7eef43f87cf6ecdc0dfd"
server_key_iv: "f11c34c9fc9d68a7b2ee62f9e261f522"
server_key_version: "encryptionkey@0"
I20260620 15:05:13.611323 22907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260620 15:05:13.615036 22922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:13.616850 22907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:13.617060 22907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da"
format_stamp: "Formatted at 2026-06-20 15:05:13 on dist-test-slave-n5vv"
server_key: "74de18f2b12b7eef43f87cf6ecdc0dfd"
server_key_iv: "f11c34c9fc9d68a7b2ee62f9e261f522"
server_key_version: "encryptionkey@0"
I20260620 15:05:13.617393 22907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:13.623274 22907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:13.627668 22907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:13.627982 22907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:13.647027 22907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33829
I20260620 15:05:13.647068 22973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33829 every 8 connection(s)
I20260620 15:05:13.648532 22907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:13.652410 22974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:13.657794 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 22907
I20260620 15:05:13.657963 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:13.658406 2328 external_mini_cluster.cc:1428] Setting key 5ef432d89b0154c569d256dcc6f627d7
I20260620 15:05:13.665037 22974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da: Bootstrap starting.
I20260620 15:05:13.668002 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:13.660403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40399 (local address 127.0.0.1:33829)
0620 15:05:13.661309 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:13.661332 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:13.661385 (+ 53us) server_negotiation.cc:407] Connection header received
0620 15:05:13.663920 (+ 2535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:13.663953 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:13.664430 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:13.664930 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:13.665973 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:13.665981 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:13.665998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:13.666015 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:13.666085 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:13.666110 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:13.666319 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:13.666451 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 15:05:13.666649 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":405,"threads_started":1}
I20260620 15:05:13.669060 22974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:13.670269 22974 log.cc:824] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:13.671921 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:13.673756 22974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da: No bootstrap required, opened a new log
I20260620 15:05:13.680020 22977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:13.677260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48355 (local address 127.0.0.1:33829)
0620 15:05:13.677547 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:13.677554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:13.677949 (+ 395us) server_negotiation.cc:407] Connection header received
0620 15:05:13.678099 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:13.678104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:13.678182 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:13.678317 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:13.679309 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:13.679315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:13.679320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:13.679341 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:13.679398 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:13.679419 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:13.679558 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:13.679649 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 15:05:13.679712 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260620 15:05:13.683512 22974 raft_consensus.cc:348] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260620 15:05:13.683924 22974 raft_consensus.cc:374] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:13.684000 22974 raft_consensus.cc:729] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c56ddc9aa0bd4ed1a2219bc69b0580da, State: Initialized, Role: FOLLOWER
I20260620 15:05:13.684603 22974 consensus_queue.cc:260] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260620 15:05:13.684824 22974 raft_consensus.cc:388] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:13.684976 22974 raft_consensus.cc:482] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:13.685115 22974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:13.687801 22974 raft_consensus.cc:504] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260620 15:05:13.688304 22974 leader_election.cc:302] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c56ddc9aa0bd4ed1a2219bc69b0580da; no voters:
I20260620 15:05:13.688856 22974 leader_election.cc:288] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:13.689028 22984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:13.689765 22984 raft_consensus.cc:686] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [term 1 LEADER]: Becoming Leader. State: Replica: c56ddc9aa0bd4ed1a2219bc69b0580da, State: Running, Role: LEADER
I20260620 15:05:13.690361 22984 consensus_queue.cc:237] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260620 15:05:13.691088 22974 sys_catalog.cc:564] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:13.693018 22986 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [sys.catalog]: SysCatalogTable state changed. Reason: New leader c56ddc9aa0bd4ed1a2219bc69b0580da. Latest consensus state: current_term: 1 leader_uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } } }
I20260620 15:05:13.693374 22986 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:13.693662 22985 sys_catalog.cc:455] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c56ddc9aa0bd4ed1a2219bc69b0580da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } } }
I20260620 15:05:13.694058 22985 sys_catalog.cc:458] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:13.694489 22989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:13.701891 22989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:13.711931 22989 catalog_manager.cc:1347] Generated new cluster ID: bf8b3c8c786f4473aea824b8f97c408a
I20260620 15:05:13.712093 22989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:13.738556 22989 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:13.739136 22989 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:13.757856 22989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c56ddc9aa0bd4ed1a2219bc69b0580da: Generated new TSK 0
I20260620 15:05:13.758785 22989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:13.768031 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22907
W20260620 15:05:13.769131 22969 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40399
W20260620 15:05:13.769834 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 22907: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:13.769995 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 22907
2026-06-20T15:05:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:13 dist-test-slave-n5vv krb5kdc[23005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:13 dist-test-slave-n5vv krb5kdc[23005](info): set up 2 sockets
Jun 20 15:05:13 dist-test-slave-n5vv krb5kdc[23005](info): commencing operation
krb5kdc: starting...
W20260620 15:05:14.901857 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:15 dist-test-slave-n5vv krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:15.160140 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:15.494742 23018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:15.495103 23018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:15.495265 23018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:15.502741 23018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:15.502911 23018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:15.503021 23018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:15.503127 23018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:15.503211 23018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:15.513418 23018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:15.514817 23018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:15.516165 23018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:15.527786 23023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:15.527868 23024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:15.527822 23026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:15.528870 23018 server_base.cc:1034] running on GCE node
I20260620 15:05:15.529704 23018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:15.531111 23018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:15.532322 23018 hybrid_clock.cc:648] HybridClock initialized: now 1781967915532284 us; error 35 us; skew 500 ppm
I20260620 15:05:15.532624 23018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:15 dist-test-slave-n5vv krb5kdc[23005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:15.535812 23018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:15.537282 23018 webserver.cc:460] Webserver started at http://127.0.0.1:44665/ using document root <none> and password file <none>
I20260620 15:05:15.538021 23018 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:15.538156 23018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:15.538460 23018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:15.541401 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "0ad5ca26342f4acfa7ff795c56b15319"
format_stamp: "Formatted at 2026-06-20 15:05:15 on dist-test-slave-n5vv"
server_key: "6d08ef1e423856ecfbc5616d14072d6b"
server_key_iv: "e0beadfd31c9f0e7d2b49558719aa26b"
server_key_version: "encryptionkey@0"
I20260620 15:05:15.542239 23018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "0ad5ca26342f4acfa7ff795c56b15319"
format_stamp: "Formatted at 2026-06-20 15:05:15 on dist-test-slave-n5vv"
server_key: "6d08ef1e423856ecfbc5616d14072d6b"
server_key_iv: "e0beadfd31c9f0e7d2b49558719aa26b"
server_key_version: "encryptionkey@0"
I20260620 15:05:15.547354 23018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:05:15.550724 23033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:15.552100 23018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:15.552322 23018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "0ad5ca26342f4acfa7ff795c56b15319"
format_stamp: "Formatted at 2026-06-20 15:05:15 on dist-test-slave-n5vv"
server_key: "6d08ef1e423856ecfbc5616d14072d6b"
server_key_iv: "e0beadfd31c9f0e7d2b49558719aa26b"
server_key_version: "encryptionkey@0"
I20260620 15:05:15.552515 23018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:15.575574 23018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:15.580286 23018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:15.580613 23018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:15.599934 23018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45379
I20260620 15:05:15.599957 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45379 every 8 connection(s)
I20260620 15:05:15.601424 23018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:15.605379 23085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:15.605451 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23018
I20260620 15:05:15.605657 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:15.606086 2328 external_mini_cluster.cc:1428] Setting key 4722c53468127cc6d1ef4b473e2d0741
I20260620 15:05:15.620188 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319: Bootstrap starting.
Jun 20 15:05:15 dist-test-slave-n5vv krb5kdc[23005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:05:15.624112 23085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:15.625501 23085 log.cc:824] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:15.629081 23085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319: No bootstrap required, opened a new log
I20260620 15:05:15.630810 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:15.607914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56263 (local address 127.0.0.1:45379)
0620 15:05:15.608798 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:15.608814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:15.608870 (+ 56us) server_negotiation.cc:407] Connection header received
0620 15:05:15.610712 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:15.610740 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:15.611283 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:15.611729 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:15.612980 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:15.615014 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:15.616293 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:15.617407 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:15.617625 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:15.621828 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:15.621840 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:15.621861 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:15.621919 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:15.625908 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:15.626925 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:15.626933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:15.626938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:15.627254 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:15.627617 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:15.627621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:15.627624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:15.628320 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:15.628618 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:15.628976 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 15:05:15.629435 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":375,"threads_started":1}
I20260620 15:05:15.634769 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:15.637874 23085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5ca26342f4acfa7ff795c56b15319" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260620 15:05:15.638226 23085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:15.638317 23085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad5ca26342f4acfa7ff795c56b15319, State: Initialized, Role: FOLLOWER
I20260620 15:05:15.639006 23085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5ca26342f4acfa7ff795c56b15319" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260620 15:05:15.639320 23085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:15.639467 23085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:15.639640 23085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:15.642772 23085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5ca26342f4acfa7ff795c56b15319" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260620 15:05:15.643380 23085 leader_election.cc:302] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ad5ca26342f4acfa7ff795c56b15319; no voters:
I20260620 15:05:15.644028 23085 leader_election.cc:288] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:15.644130 23095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:15.646111 23085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:15.649358 23095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [term 1 LEADER]: Becoming Leader. State: Replica: 0ad5ca26342f4acfa7ff795c56b15319, State: Running, Role: LEADER
I20260620 15:05:15.650077 23095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5ca26342f4acfa7ff795c56b15319" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } }
I20260620 15:05:15.655154 23097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ad5ca26342f4acfa7ff795c56b15319. Latest consensus state: current_term: 1 leader_uuid: "0ad5ca26342f4acfa7ff795c56b15319" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5ca26342f4acfa7ff795c56b15319" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } } }
I20260620 15:05:15.655552 23097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:15.655417 23096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ad5ca26342f4acfa7ff795c56b15319" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ad5ca26342f4acfa7ff795c56b15319" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45379 } } }
I20260620 15:05:15.655735 23096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:15.657038 23109 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:15.658533 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:15.639199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54225 (local address 127.0.0.1:45379)
0620 15:05:15.639427 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:15.639433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:15.639796 (+ 363us) server_negotiation.cc:407] Connection header received
0620 15:05:15.639974 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:15.639977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:15.640037 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:15.640113 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:15.641946 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:15.642646 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:15.644481 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:15.645137 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:15.645267 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:15.649464 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:15.649498 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:15.649518 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:15.649577 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:15.654020 (+ 4443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:15.655018 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:15.655024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:15.655028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:15.655157 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:15.655651 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:15.655658 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:15.655661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:15.656585 (+ 924us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:15.657880 (+ 1295us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:15.658035 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 15:05:15.658294 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:05:15.662674 23109 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:15.671620 23109 catalog_manager.cc:1347] Generated new cluster ID: cc9c742a473a40eba571b05990fd832b
I20260620 15:05:15.671748 23109 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:15.685312 23109 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:15.687247 23109 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:15.696709 23109 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ad5ca26342f4acfa7ff795c56b15319: Generated new TSK 0
I20260620 15:05:15.697744 23109 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:15.709966 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23018
I20260620 15:05:15.711578 23080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:15.802661 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23018
2026-06-20T15:05:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2034 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:15 dist-test-slave-n5vv krb5kdc[23117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:15 dist-test-slave-n5vv krb5kdc[23117](info): set up 2 sockets
Jun 20 15:05:15 dist-test-slave-n5vv krb5kdc[23117](info): commencing operation
krb5kdc: starting...
W20260620 15:05:16.918002 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:17 dist-test-slave-n5vv krb5kdc[23117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:17.171022 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38625
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:17.501991 23130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:17.502350 23130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:17.502523 23130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:17.510040 23130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:17.510211 23130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:17.510317 23130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:17.510432 23130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:17.510510 23130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:17.520635 23130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:17.522099 23130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:17.523442 23130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:17.535487 23136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:17.535498 23135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:17.536098 23138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:17.536293 23130 server_base.cc:1034] running on GCE node
I20260620 15:05:17.537132 23130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:17.538556 23130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:17.539808 23130 hybrid_clock.cc:648] HybridClock initialized: now 1781967917539745 us; error 46 us; skew 500 ppm
I20260620 15:05:17.540191 23130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:17 dist-test-slave-n5vv krb5kdc[23117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:17.543512 23130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:17.544939 23130 webserver.cc:460] Webserver started at http://127.0.0.1:46133/ using document root <none> and password file <none>
I20260620 15:05:17.545756 23130 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:17.545902 23130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:17.546201 23130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:17.549150 23130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "c0dca7d4e1d94738966ad063bb10152c"
format_stamp: "Formatted at 2026-06-20 15:05:17 on dist-test-slave-n5vv"
server_key: "6df1e7c5d821f59ee5bb934caf670844"
server_key_iv: "d519b84fae3b21c2f1cb59dbc92e115f"
server_key_version: "encryptionkey@0"
I20260620 15:05:17.549973 23130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "c0dca7d4e1d94738966ad063bb10152c"
format_stamp: "Formatted at 2026-06-20 15:05:17 on dist-test-slave-n5vv"
server_key: "6df1e7c5d821f59ee5bb934caf670844"
server_key_iv: "d519b84fae3b21c2f1cb59dbc92e115f"
server_key_version: "encryptionkey@0"
I20260620 15:05:17.555212 23130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 15:05:17.558980 23145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:17.560527 23130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:17.560734 23130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "c0dca7d4e1d94738966ad063bb10152c"
format_stamp: "Formatted at 2026-06-20 15:05:17 on dist-test-slave-n5vv"
server_key: "6df1e7c5d821f59ee5bb934caf670844"
server_key_iv: "d519b84fae3b21c2f1cb59dbc92e115f"
server_key_version: "encryptionkey@0"
I20260620 15:05:17.560946 23130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:17.579623 23130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:17.583856 23130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:17.584146 23130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:17.602991 23130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38625
I20260620 15:05:17.603024 23196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38625 every 8 connection(s)
I20260620 15:05:17.604377 23130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:17.604821 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23130
I20260620 15:05:17.605024 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:17.605685 2328 external_mini_cluster.cc:1428] Setting key 47dbcdeff20bdfb4cf91b966854d226e
I20260620 15:05:17.608775 23197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:17.618486 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:17.607473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44637 (local address 127.0.0.1:38625)
0620 15:05:17.608231 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:17.608244 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:17.608292 (+ 48us) server_negotiation.cc:407] Connection header received
0620 15:05:17.610015 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:17.610037 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:17.610406 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:17.610796 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:17.611349 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:17.612627 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:17.614033 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:17.614707 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:17.614908 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:17.616118 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:17.616126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:17.616139 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:17.616688 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:17.616777 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:17.616805 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:17.616956 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:17.617182 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 15:05:17.617449 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":352,"threads_started":1}
I20260620 15:05:17.619656 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c: Bootstrap starting.
I20260620 15:05:17.622264 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:17.623041 23197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:17.624022 23197 log.cc:824] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:17.627440 23197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c: No bootstrap required, opened a new log
I20260620 15:05:17.633109 23200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:17.626865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54055 (local address 127.0.0.1:38625)
0620 15:05:17.627016 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:17.627021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:17.627541 (+ 520us) server_negotiation.cc:407] Connection header received
0620 15:05:17.627810 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:17.627814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:17.627878 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:17.627995 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:17.628497 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:17.629237 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:17.630655 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:17.631115 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:17.631253 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:17.632341 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:17.632346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:17.632350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:17.632372 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:17.632411 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:17.632429 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:17.632534 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:17.632678 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 15:05:17.632777 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 15:05:17.635601 23197 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dca7d4e1d94738966ad063bb10152c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38625 } }
I20260620 15:05:17.636016 23197 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:17.636123 23197 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0dca7d4e1d94738966ad063bb10152c, State: Initialized, Role: FOLLOWER
I20260620 15:05:17.636695 23197 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dca7d4e1d94738966ad063bb10152c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38625 } }
I20260620 15:05:17.636955 23197 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:17.637099 23197 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:17.637277 23197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:17.639873 23197 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dca7d4e1d94738966ad063bb10152c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38625 } }
I20260620 15:05:17.640452 23197 leader_election.cc:302] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0dca7d4e1d94738966ad063bb10152c; no voters:
I20260620 15:05:17.641002 23197 leader_election.cc:288] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:17.641176 23207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:17.641909 23207 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [term 1 LEADER]: Becoming Leader. State: Replica: c0dca7d4e1d94738966ad063bb10152c, State: Running, Role: LEADER
I20260620 15:05:17.642479 23207 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dca7d4e1d94738966ad063bb10152c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38625 } }
I20260620 15:05:17.643293 23197 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:17.645576 23209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0dca7d4e1d94738966ad063bb10152c. Latest consensus state: current_term: 1 leader_uuid: "c0dca7d4e1d94738966ad063bb10152c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dca7d4e1d94738966ad063bb10152c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38625 } } }
I20260620 15:05:17.645833 23208 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0dca7d4e1d94738966ad063bb10152c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0dca7d4e1d94738966ad063bb10152c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38625 } } }
I20260620 15:05:17.646083 23209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:17.646140 23208 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c [sys.catalog]: This master's current role is: LEADER
W20260620 15:05:17.657330 23223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:05:17.657500 23223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:05:17.657661 23215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:17.659451 23215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:17.668588 23215 catalog_manager.cc:1347] Generated new cluster ID: 73cdf1f07db84129b1531672cc32e0e2
I20260620 15:05:17.668733 23215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:17.691323 23215 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:17.692852 23215 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:17.711154 23215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0dca7d4e1d94738966ad063bb10152c: Generated new TSK 0
I20260620 15:05:17.712296 23215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:17.718595 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23130
W20260620 15:05:17.720068 23192 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44637
W20260620 15:05:17.720795 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23130: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:17.720952 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23130
2026-06-20T15:05:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:17 dist-test-slave-n5vv krb5kdc[23228](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:17 dist-test-slave-n5vv krb5kdc[23228](info): set up 2 sockets
Jun 20 15:05:17 dist-test-slave-n5vv krb5kdc[23228](info): commencing operation
krb5kdc: starting...
W20260620 15:05:18.857879 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:19 dist-test-slave-n5vv krb5kdc[23228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:19.110466 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:19.453732 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:19.454079 23241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:19.454241 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:19.461624 23241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:19.461786 23241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:19.461896 23241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:19.462004 23241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:19.462085 23241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:19.471827 23241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:19.473151 23241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:19.474505 23241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:19.487041 23249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:19.487138 23246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:19.487152 23247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:19.488152 23241 server_base.cc:1034] running on GCE node
I20260620 15:05:19.488957 23241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:19.490548 23241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:19.491758 23241 hybrid_clock.cc:648] HybridClock initialized: now 1781967919491707 us; error 42 us; skew 500 ppm
I20260620 15:05:19.492116 23241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:19 dist-test-slave-n5vv krb5kdc[23228](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:19.500013 23241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:19.501730 23241 webserver.cc:460] Webserver started at http://127.0.0.1:45205/ using document root <none> and password file <none>
I20260620 15:05:19.502449 23241 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:19.502578 23241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:19.502892 23241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:19.505918 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "8c73f0bba7a94557abb437e02a64111a"
format_stamp: "Formatted at 2026-06-20 15:05:19 on dist-test-slave-n5vv"
server_key: "c5dbc996168ba3a6cdba046064f8b985"
server_key_iv: "cbfabb8be5ecccbdc7521074aa640f62"
server_key_version: "encryptionkey@0"
I20260620 15:05:19.506676 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "8c73f0bba7a94557abb437e02a64111a"
format_stamp: "Formatted at 2026-06-20 15:05:19 on dist-test-slave-n5vv"
server_key: "c5dbc996168ba3a6cdba046064f8b985"
server_key_iv: "cbfabb8be5ecccbdc7521074aa640f62"
server_key_version: "encryptionkey@0"
I20260620 15:05:19.512188 23241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260620 15:05:19.515720 23256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:19.517113 23241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 15:05:19.517300 23241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "8c73f0bba7a94557abb437e02a64111a"
format_stamp: "Formatted at 2026-06-20 15:05:19 on dist-test-slave-n5vv"
server_key: "c5dbc996168ba3a6cdba046064f8b985"
server_key_iv: "cbfabb8be5ecccbdc7521074aa640f62"
server_key_version: "encryptionkey@0"
I20260620 15:05:19.517524 23241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:19.523197 23241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:19.527124 23241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:19.527420 23241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:19.545709 23241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40753
I20260620 15:05:19.545751 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40753 every 8 connection(s)
I20260620 15:05:19.547202 23241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:19.551357 23308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:19.553778 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23241
I20260620 15:05:19.554003 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:19.554504 2328 external_mini_cluster.cc:1428] Setting key eff1e3bc3ca1898ce7902e4a4ed293af
I20260620 15:05:19.562851 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:19.556476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50129 (local address 127.0.0.1:40753)
0620 15:05:19.557363 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:19.557376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:19.557417 (+ 41us) server_negotiation.cc:407] Connection header received
0620 15:05:19.559251 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:19.559273 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:19.559677 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:19.560026 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:19.561159 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:19.561166 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:19.561180 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:19.561195 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:19.561257 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:19.561278 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:19.561457 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:19.561708 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 15:05:19.561919 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":427,"threads_started":1}
I20260620 15:05:19.563261 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a: Bootstrap starting.
I20260620 15:05:19.566562 23308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:19.566797 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:19.567641 23308 log.cc:824] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:19.570474 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a: No bootstrap required, opened a new log
I20260620 15:05:19.574043 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:19.571426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45967 (local address 127.0.0.1:40753)
0620 15:05:19.571602 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:19.571608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:19.572056 (+ 448us) server_negotiation.cc:407] Connection header received
0620 15:05:19.572179 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:19.572185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:19.572245 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:19.572355 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:19.573328 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:19.573336 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:19.573345 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:19.573362 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:19.573413 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:19.573432 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:19.573572 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:19.573687 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 15:05:19.573754 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 15:05:19.578892 23308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c73f0bba7a94557abb437e02a64111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260620 15:05:19.579237 23308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:19.579317 23308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c73f0bba7a94557abb437e02a64111a, State: Initialized, Role: FOLLOWER
I20260620 15:05:19.579838 23308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c73f0bba7a94557abb437e02a64111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260620 15:05:19.580157 23308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:19.580331 23308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:19.580526 23308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:19.582837 23308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c73f0bba7a94557abb437e02a64111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260620 15:05:19.583297 23308 leader_election.cc:302] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c73f0bba7a94557abb437e02a64111a; no voters:
I20260620 15:05:19.583849 23308 leader_election.cc:288] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:19.584007 23318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:19.584649 23318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [term 1 LEADER]: Becoming Leader. State: Replica: 8c73f0bba7a94557abb437e02a64111a, State: Running, Role: LEADER
I20260620 15:05:19.585223 23318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c73f0bba7a94557abb437e02a64111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } }
I20260620 15:05:19.585991 23308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:19.588328 23319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c73f0bba7a94557abb437e02a64111a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c73f0bba7a94557abb437e02a64111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } } }
I20260620 15:05:19.588307 23320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c73f0bba7a94557abb437e02a64111a. Latest consensus state: current_term: 1 leader_uuid: "8c73f0bba7a94557abb437e02a64111a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c73f0bba7a94557abb437e02a64111a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40753 } } }
I20260620 15:05:19.588810 23320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:19.588810 23319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:19.590684 23323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:19.597254 23323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:19.606899 23323 catalog_manager.cc:1347] Generated new cluster ID: 5a6c15bf94794ac8b80aa888c84f55ca
I20260620 15:05:19.607040 23323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:19.623178 23323 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:19.623616 23323 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:19.635129 23323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c73f0bba7a94557abb437e02a64111a: Generated new TSK 0
I20260620 15:05:19.635946 23323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:19.642163 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23241
W20260620 15:05:19.643740 23303 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50129
W20260620 15:05:19.644451 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23241: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:19.644603 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23241
2026-06-20T15:05:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:19 dist-test-slave-n5vv krb5kdc[23339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:19 dist-test-slave-n5vv krb5kdc[23339](info): set up 2 sockets
Jun 20 15:05:19 dist-test-slave-n5vv krb5kdc[23339](info): commencing operation
krb5kdc: starting...
W20260620 15:05:20.765849 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:20 dist-test-slave-n5vv krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:21.020821 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40685
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:21.369871 23352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:21.370326 23352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:21.370505 23352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:21.377970 23352 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:21.378137 23352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:21.378244 23352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:21.378356 23352 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:21.378443 23352 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:21.388378 23352 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:21.389756 23352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:21.391110 23352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:21.403149 23357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:21.403182 23358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:21.403124 23360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:21.404417 23352 server_base.cc:1034] running on GCE node
I20260620 15:05:21.405283 23352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:21.406728 23352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:21.407938 23352 hybrid_clock.cc:648] HybridClock initialized: now 1781967921407876 us; error 57 us; skew 500 ppm
I20260620 15:05:21.408295 23352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:21 dist-test-slave-n5vv krb5kdc[23339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:21.411506 23352 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:21.412889 23352 webserver.cc:460] Webserver started at http://127.0.0.1:34795/ using document root <none> and password file <none>
I20260620 15:05:21.413604 23352 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:21.413704 23352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:21.414027 23352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:21.416954 23352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b"
format_stamp: "Formatted at 2026-06-20 15:05:21 on dist-test-slave-n5vv"
server_key: "9d881e94def36a25501b08e2b778740a"
server_key_iv: "0acebe6ceb357b0c33280a019dd7e504"
server_key_version: "encryptionkey@0"
I20260620 15:05:21.417788 23352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b"
format_stamp: "Formatted at 2026-06-20 15:05:21 on dist-test-slave-n5vv"
server_key: "9d881e94def36a25501b08e2b778740a"
server_key_iv: "0acebe6ceb357b0c33280a019dd7e504"
server_key_version: "encryptionkey@0"
I20260620 15:05:21.423030 23352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260620 15:05:21.426595 23367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:21.428004 23352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:05:21.428200 23352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b"
format_stamp: "Formatted at 2026-06-20 15:05:21 on dist-test-slave-n5vv"
server_key: "9d881e94def36a25501b08e2b778740a"
server_key_iv: "0acebe6ceb357b0c33280a019dd7e504"
server_key_version: "encryptionkey@0"
I20260620 15:05:21.428423 23352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:21.443748 23352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:21.448443 23352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:21.448752 23352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:21.467792 23352 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40685
I20260620 15:05:21.467826 23418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40685 every 8 connection(s)
I20260620 15:05:21.469168 23352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:21.473045 23419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:21.475139 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23352
I20260620 15:05:21.475383 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:21.475840 2328 external_mini_cluster.cc:1428] Setting key b7a234bef4d9400f7a3122c89d525e20
I20260620 15:05:21.486506 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b: Bootstrap starting.
I20260620 15:05:21.490078 23419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:21.491083 23419 log.cc:824] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b: Log is configured to *not* fsync() on all Append() calls
Jun 20 15:05:21 dist-test-slave-n5vv krb5kdc[23339](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:05:21.494405 23419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b: No bootstrap required, opened a new log
I20260620 15:05:21.499989 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:21.477850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:53367 (local address 127.0.0.1:40685)
0620 15:05:21.478647 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:21.478663 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:21.478714 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:05:21.481233 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:21.481264 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:21.481799 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:21.482242 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:21.483492 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:21.485598 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:21.486972 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:21.487958 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:21.488170 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:21.492483 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:21.492490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:21.492504 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:21.492545 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:21.495679 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:21.496485 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:21.496491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:21.496501 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:21.496684 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:21.497160 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:21.497164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:21.497166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:21.497809 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:21.498124 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:21.498645 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 15:05:21.498966 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":324,"threads_started":1}
I20260620 15:05:21.502344 23419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 15:05:21.502801 23419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:21.502929 23419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75ca5eee7cbe48c3a637a5b5fafbe45b, State: Initialized, Role: FOLLOWER
I20260620 15:05:21.503492 23419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 15:05:21.503747 23419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:21.503903 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:21.503904 23419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:21.504069 23419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:21.506388 23419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 15:05:21.506958 23419 leader_election.cc:302] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75ca5eee7cbe48c3a637a5b5fafbe45b; no voters:
I20260620 15:05:21.507620 23419 leader_election.cc:288] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:21.507731 23428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:21.510274 23419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:21.518673 23428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [term 1 LEADER]: Becoming Leader. State: Replica: 75ca5eee7cbe48c3a637a5b5fafbe45b, State: Running, Role: LEADER
I20260620 15:05:21.519933 23428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } }
I20260620 15:05:21.523327 23430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75ca5eee7cbe48c3a637a5b5fafbe45b. Latest consensus state: current_term: 1 leader_uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } } }
I20260620 15:05:21.523800 23430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:21.524737 23429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75ca5eee7cbe48c3a637a5b5fafbe45b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40685 } } }
I20260620 15:05:21.525025 23429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:21.538131 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:21.508824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:43163 (local address 127.0.0.1:40685)
0620 15:05:21.509041 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:21.509050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:21.518217 (+ 9167us) server_negotiation.cc:407] Connection header received
0620 15:05:21.518437 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:21.518453 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:21.518568 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:21.518701 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:21.520922 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:21.521903 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:21.523730 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:21.524808 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:21.524959 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:21.529089 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:21.529099 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:21.529107 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:21.529159 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:21.533718 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:21.535025 (+ 1307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:21.535034 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:21.535040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:21.535206 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:21.535899 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:21.535907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:21.535913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:21.536395 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:21.536642 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:21.537526 (+ 884us) server_negotiation.cc:299] Negotiation successful
0620 15:05:21.537764 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:05:21.538971 23439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:21.545115 23439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:21.554776 23439 catalog_manager.cc:1347] Generated new cluster ID: ff60b73f261a448cb0eb2709d6b84fb7
I20260620 15:05:21.554908 23439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:21.569010 23439 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:21.570327 23439 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:21.589038 23439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75ca5eee7cbe48c3a637a5b5fafbe45b: Generated new TSK 0
I20260620 15:05:21.590016 23439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:21.606870 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23352
I20260620 15:05:21.608003 23414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:21.703440 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23352
W20260620 15:05:21.722299 23337 connection.cc:535] client connection to 127.0.0.1:40685 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40685): Connection reset by peer (error 104)
2026-06-20T15:05:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2060 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:21 dist-test-slave-n5vv krb5kdc[23452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:21 dist-test-slave-n5vv krb5kdc[23452](info): set up 2 sockets
Jun 20 15:05:21 dist-test-slave-n5vv krb5kdc[23452](info): commencing operation
krb5kdc: starting...
W20260620 15:05:22.817847 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:22 dist-test-slave-n5vv krb5kdc[23452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:23.068722 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:23.394955 23465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:23.395273 23465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:23.395370 23465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:23.402830 23465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:23.402961 23465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:23.403009 23465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:23.403067 23465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:23.403115 23465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:23.413362 23465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:23.414728 23465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:23.416030 23465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:23.428462 23470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:23.428488 23473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:23.428443 23471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:23.429455 23465 server_base.cc:1034] running on GCE node
I20260620 15:05:23.430357 23465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:23.431917 23465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:23.433137 23465 hybrid_clock.cc:648] HybridClock initialized: now 1781967923433087 us; error 52 us; skew 500 ppm
I20260620 15:05:23.433630 23465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:23 dist-test-slave-n5vv krb5kdc[23452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:23.437091 23465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:23.438889 23465 webserver.cc:460] Webserver started at http://127.0.0.1:39503/ using document root <none> and password file <none>
I20260620 15:05:23.439646 23465 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:23.439800 23465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:23.440107 23465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:23.443277 23465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d"
format_stamp: "Formatted at 2026-06-20 15:05:23 on dist-test-slave-n5vv"
server_key: "2bed36da66fc1d027efb74063a98a5d0"
server_key_iv: "07f1e244c7ccc6fda66d721881bf281a"
server_key_version: "encryptionkey@0"
I20260620 15:05:23.444058 23465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d"
format_stamp: "Formatted at 2026-06-20 15:05:23 on dist-test-slave-n5vv"
server_key: "2bed36da66fc1d027efb74063a98a5d0"
server_key_iv: "07f1e244c7ccc6fda66d721881bf281a"
server_key_version: "encryptionkey@0"
I20260620 15:05:23.449568 23465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:05:23.453197 23480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:23.454732 23465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:05:23.454950 23465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d"
format_stamp: "Formatted at 2026-06-20 15:05:23 on dist-test-slave-n5vv"
server_key: "2bed36da66fc1d027efb74063a98a5d0"
server_key_iv: "07f1e244c7ccc6fda66d721881bf281a"
server_key_version: "encryptionkey@0"
I20260620 15:05:23.455134 23465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:23.474195 23465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:23.478765 23465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:23.479022 23465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:23.498394 23465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35265
I20260620 15:05:23.498486 23531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35265 every 8 connection(s)
I20260620 15:05:23.499825 23465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:23.502604 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23465
I20260620 15:05:23.502820 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:23.503415 2328 external_mini_cluster.cc:1428] Setting key 01c71cf04cd6372854d15e2c10b28ffa
I20260620 15:05:23.504798 23532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:23.516569 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:23.505371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:52405 (local address 127.0.0.1:35265)
0620 15:05:23.506182 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:23.506194 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:23.506272 (+ 78us) server_negotiation.cc:407] Connection header received
0620 15:05:23.507944 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:23.507964 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:23.508381 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:23.508779 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:23.509319 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:23.510708 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:23.512079 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:23.512748 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:23.512940 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:23.514310 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:23.514321 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:23.514335 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:23.514898 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:23.514971 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:23.514996 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:23.515180 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:23.515445 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 15:05:23.515623 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":339,"threads_started":1}
I20260620 15:05:23.517982 23532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d: Bootstrap starting.
I20260620 15:05:23.520481 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:23.521674 23532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:23.522744 23532 log.cc:824] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:23.526479 23532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d: No bootstrap required, opened a new log
I20260620 15:05:23.532891 23535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:23.524898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:39121 (local address 127.0.0.1:35265)
0620 15:05:23.525094 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:23.525101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:23.526096 (+ 995us) server_negotiation.cc:407] Connection header received
0620 15:05:23.526327 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:23.526331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:23.526392 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:23.526476 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:23.527095 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:23.527864 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:23.530125 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:23.530608 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:23.530737 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:23.532057 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:23.532062 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:23.532066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:23.532083 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:23.532128 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:23.532146 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:23.532238 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:23.532451 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 15:05:23.532496 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:05:23.536571 23532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35265 } }
I20260620 15:05:23.536998 23532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:23.537130 23532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b9b4195489a44c1a5a3c1cd9e3bd37d, State: Initialized, Role: FOLLOWER
I20260620 15:05:23.537894 23532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35265 } }
I20260620 15:05:23.538224 23532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:23.538396 23532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:23.538587 23532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:23.541438 23532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35265 } }
I20260620 15:05:23.542091 23532 leader_election.cc:302] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b9b4195489a44c1a5a3c1cd9e3bd37d; no voters:
I20260620 15:05:23.542711 23532 leader_election.cc:288] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:23.542847 23542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:23.543639 23542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [term 1 LEADER]: Becoming Leader. State: Replica: 7b9b4195489a44c1a5a3c1cd9e3bd37d, State: Running, Role: LEADER
I20260620 15:05:23.544224 23542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35265 } }
I20260620 15:05:23.544973 23532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:23.547465 23544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b9b4195489a44c1a5a3c1cd9e3bd37d. Latest consensus state: current_term: 1 leader_uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35265 } } }
I20260620 15:05:23.547829 23544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:23.547524 23543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b9b4195489a44c1a5a3c1cd9e3bd37d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35265 } } }
I20260620 15:05:23.547988 23543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:23.550424 23549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:23.555670 23549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:23.565919 23549 catalog_manager.cc:1347] Generated new cluster ID: 2326cc94a6814d5fb8ab2460f2d326fc
I20260620 15:05:23.566097 23549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:23.602427 23549 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:23.603826 23549 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:23.617080 23549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b9b4195489a44c1a5a3c1cd9e3bd37d: Generated new TSK 0
I20260620 15:05:23.618081 23549 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:23.630043 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23465
W20260620 15:05:23.631390 23527 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.44:52405
W20260620 15:05:23.632172 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23465: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:23.632334 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23465
W20260620 15:05:23.649125 23450 connection.cc:535] client connection to 127.0.0.1:35265 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35265): Connection reset by peer (error 104)
2026-06-20T15:05:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:23 dist-test-slave-n5vv krb5kdc[23563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:23 dist-test-slave-n5vv krb5kdc[23563](info): set up 2 sockets
Jun 20 15:05:23 dist-test-slave-n5vv krb5kdc[23563](info): commencing operation
krb5kdc: starting...
W20260620 15:05:24.750028 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:24 dist-test-slave-n5vv krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:25.004880 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:25.335060 23576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:25.335425 23576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:25.335587 23576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:25.342810 23576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:25.342973 23576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:25.343072 23576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:25.343174 23576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:25.343247 23576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:25.353080 23576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:25.354506 23576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:25.355867 23576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:25.368137 23582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:25.368212 23584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:25.368170 23581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:25.369266 23576 server_base.cc:1034] running on GCE node
I20260620 15:05:25.370169 23576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:25.371640 23576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:25.372884 23576 hybrid_clock.cc:648] HybridClock initialized: now 1781967925372830 us; error 50 us; skew 500 ppm
I20260620 15:05:25.373246 23576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:25 dist-test-slave-n5vv krb5kdc[23563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:25.379653 23576 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:25.381244 23576 webserver.cc:460] Webserver started at http://127.0.0.1:42437/ using document root <none> and password file <none>
I20260620 15:05:25.382028 23576 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:25.382179 23576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:25.382494 23576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:25.385449 23576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "4741df1d220e487c8baf22409c088e77"
format_stamp: "Formatted at 2026-06-20 15:05:25 on dist-test-slave-n5vv"
server_key: "4af7e1977353cfcaad89b91d095efb4b"
server_key_iv: "49d9cb9be537f9bd6d39ab224305fd09"
server_key_version: "encryptionkey@0"
I20260620 15:05:25.386298 23576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "4741df1d220e487c8baf22409c088e77"
format_stamp: "Formatted at 2026-06-20 15:05:25 on dist-test-slave-n5vv"
server_key: "4af7e1977353cfcaad89b91d095efb4b"
server_key_iv: "49d9cb9be537f9bd6d39ab224305fd09"
server_key_version: "encryptionkey@0"
I20260620 15:05:25.391295 23576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:05:25.394682 23591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:25.395953 23576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:25.396138 23576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "4741df1d220e487c8baf22409c088e77"
format_stamp: "Formatted at 2026-06-20 15:05:25 on dist-test-slave-n5vv"
server_key: "4af7e1977353cfcaad89b91d095efb4b"
server_key_iv: "49d9cb9be537f9bd6d39ab224305fd09"
server_key_version: "encryptionkey@0"
I20260620 15:05:25.396340 23576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:25.402066 23576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:25.406056 23576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:25.406350 23576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:25.424853 23576 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42875
I20260620 15:05:25.424911 23642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42875 every 8 connection(s)
I20260620 15:05:25.426357 23576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:25.428994 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23576
I20260620 15:05:25.429175 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:25.429687 2328 external_mini_cluster.cc:1428] Setting key 60ddcbbd5979e5e087a393372374d161
I20260620 15:05:25.430342 23643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:25.437222 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:25.431438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:43173 (local address 127.0.0.1:42875)
0620 15:05:25.432221 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:25.432240 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:25.432280 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:05:25.433888 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:25.433909 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:25.434272 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:25.434609 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:25.435651 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:25.435657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:25.435669 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:25.435692 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:25.435762 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:25.435781 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:25.435950 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:25.436056 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 15:05:25.436288 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":352,"threads_started":1}
I20260620 15:05:25.440797 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:25.444430 23643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77: Bootstrap starting.
I20260620 15:05:25.448475 23643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:25.448510 23646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:25.446016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:37385 (local address 127.0.0.1:42875)
0620 15:05:25.446228 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:25.446235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:25.446669 (+ 434us) server_negotiation.cc:407] Connection header received
0620 15:05:25.446811 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:25.446816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:25.446874 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:25.446977 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:25.447856 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:25.447861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:25.447865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:25.447883 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:25.447928 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:25.447947 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:25.448064 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:25.448114 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 15:05:25.448171 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:05:25.449811 23643 log.cc:824] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:25.452420 23643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77: No bootstrap required, opened a new log
I20260620 15:05:25.459872 23643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741df1d220e487c8baf22409c088e77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } }
I20260620 15:05:25.460275 23643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:25.460378 23643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4741df1d220e487c8baf22409c088e77, State: Initialized, Role: FOLLOWER
I20260620 15:05:25.461097 23643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741df1d220e487c8baf22409c088e77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } }
I20260620 15:05:25.461436 23643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:25.461623 23643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:25.461812 23643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:25.463996 23643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741df1d220e487c8baf22409c088e77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } }
I20260620 15:05:25.464547 23643 leader_election.cc:302] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4741df1d220e487c8baf22409c088e77; no voters:
I20260620 15:05:25.465142 23643 leader_election.cc:288] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:25.465261 23653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:25.466001 23653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [term 1 LEADER]: Becoming Leader. State: Replica: 4741df1d220e487c8baf22409c088e77, State: Running, Role: LEADER
I20260620 15:05:25.466517 23653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741df1d220e487c8baf22409c088e77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } }
I20260620 15:05:25.467259 23643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:25.469355 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4741df1d220e487c8baf22409c088e77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741df1d220e487c8baf22409c088e77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } } }
I20260620 15:05:25.469883 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:25.470253 23655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4741df1d220e487c8baf22409c088e77. Latest consensus state: current_term: 1 leader_uuid: "4741df1d220e487c8baf22409c088e77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4741df1d220e487c8baf22409c088e77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42875 } } }
I20260620 15:05:25.470579 23655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:25.470739 23660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:25.477192 23660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:25.486778 23660 catalog_manager.cc:1347] Generated new cluster ID: ab3e980b4ee847fa8ed2b584a5c5860a
I20260620 15:05:25.486961 23660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:25.516373 23660 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:25.516897 23660 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:25.529656 23660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4741df1d220e487c8baf22409c088e77: Generated new TSK 0
I20260620 15:05:25.530575 23660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:25.547091 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23576
W20260620 15:05:25.548144 23638 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.44:43173
W20260620 15:05:25.548842 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:25.548985 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23576
2026-06-20T15:05:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:25 dist-test-slave-n5vv krb5kdc[23674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:25 dist-test-slave-n5vv krb5kdc[23674](info): set up 2 sockets
Jun 20 15:05:25 dist-test-slave-n5vv krb5kdc[23674](info): commencing operation
krb5kdc: starting...
W20260620 15:05:26.658438 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:26 dist-test-slave-n5vv krb5kdc[23674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:26.921417 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:27.261763 23687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:27.262046 23687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:27.262138 23687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:27.269779 23687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:27.269886 23687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:27.269932 23687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:27.269994 23687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:27.270030 23687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:27.279852 23687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:27.281213 23687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:27.283020 23687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:27.295068 23693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:27.295164 23695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:27.295181 23692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:27.296394 23687 server_base.cc:1034] running on GCE node
I20260620 15:05:27.297299 23687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:27.298892 23687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:27.300130 23687 hybrid_clock.cc:648] HybridClock initialized: now 1781967927300087 us; error 57 us; skew 500 ppm
I20260620 15:05:27.300487 23687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:27 dist-test-slave-n5vv krb5kdc[23674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:27.303785 23687 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:27.305343 23687 webserver.cc:460] Webserver started at http://127.0.0.1:38335/ using document root <none> and password file <none>
I20260620 15:05:27.306102 23687 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:27.306205 23687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:27.306465 23687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:27.309475 23687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "b3c878cfc75c445c97d5a171e53cde86"
format_stamp: "Formatted at 2026-06-20 15:05:27 on dist-test-slave-n5vv"
server_key: "69903ef07369609077b634001b6046f8"
server_key_iv: "134401eba27407ef9e6840c8242e9a46"
server_key_version: "encryptionkey@0"
I20260620 15:05:27.310297 23687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "b3c878cfc75c445c97d5a171e53cde86"
format_stamp: "Formatted at 2026-06-20 15:05:27 on dist-test-slave-n5vv"
server_key: "69903ef07369609077b634001b6046f8"
server_key_iv: "134401eba27407ef9e6840c8242e9a46"
server_key_version: "encryptionkey@0"
I20260620 15:05:27.315543 23687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:05:27.319164 23702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:27.320560 23687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:05:27.320771 23687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "b3c878cfc75c445c97d5a171e53cde86"
format_stamp: "Formatted at 2026-06-20 15:05:27 on dist-test-slave-n5vv"
server_key: "69903ef07369609077b634001b6046f8"
server_key_iv: "134401eba27407ef9e6840c8242e9a46"
server_key_version: "encryptionkey@0"
I20260620 15:05:27.320953 23687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:27.340348 23687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:27.344414 23687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:27.344707 23687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:27.363412 23687 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43675
I20260620 15:05:27.363421 23753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43675 every 8 connection(s)
I20260620 15:05:27.364864 23687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:27.366374 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23687
I20260620 15:05:27.366554 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:27.367091 2328 external_mini_cluster.cc:1428] Setting key 43ba14da59434aba5d9c1e2a314a6cd2
I20260620 15:05:27.369632 23754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:05:27 dist-test-slave-n5vv krb5kdc[23674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 15:05:27.382858 23754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86: Bootstrap starting.
I20260620 15:05:27.386196 23754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:27.387297 23754 log.cc:824] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:27.390581 23754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86: No bootstrap required, opened a new log
I20260620 15:05:27.391182 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:27.368923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:36739 (local address 127.0.0.1:43675)
0620 15:05:27.369762 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:27.369777 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:27.369829 (+ 52us) server_negotiation.cc:407] Connection header received
0620 15:05:27.371455 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:27.371475 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:27.371871 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:27.372205 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:27.374003 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:27.375384 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:27.377227 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:27.377967 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:27.378100 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:27.383019 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:27.383033 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:27.383048 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:27.383088 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:27.386195 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:27.387103 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:27.387109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:27.387115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:27.387282 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:27.387730 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:27.387737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:27.387741 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:27.388510 (+ 769us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:27.388799 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:27.389408 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 15:05:27.389853 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":371,"threads_started":1}
I20260620 15:05:27.395336 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:27.398862 23754 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c878cfc75c445c97d5a171e53cde86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } }
I20260620 15:05:27.399231 23754 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:27.399353 23754 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c878cfc75c445c97d5a171e53cde86, State: Initialized, Role: FOLLOWER
I20260620 15:05:27.400045 23754 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c878cfc75c445c97d5a171e53cde86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } }
I20260620 15:05:27.400286 23754 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:27.400478 23754 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:27.400669 23754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:27.402945 23754 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c878cfc75c445c97d5a171e53cde86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } }
I20260620 15:05:27.403463 23754 leader_election.cc:302] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3c878cfc75c445c97d5a171e53cde86; no voters:
I20260620 15:05:27.404129 23754 leader_election.cc:288] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:27.404242 23764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:27.405377 23764 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [term 1 LEADER]: Becoming Leader. State: Replica: b3c878cfc75c445c97d5a171e53cde86, State: Running, Role: LEADER
I20260620 15:05:27.406041 23764 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c878cfc75c445c97d5a171e53cde86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } }
I20260620 15:05:27.406571 23754 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:27.408978 23766 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3c878cfc75c445c97d5a171e53cde86. Latest consensus state: current_term: 1 leader_uuid: "b3c878cfc75c445c97d5a171e53cde86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c878cfc75c445c97d5a171e53cde86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } } }
I20260620 15:05:27.409410 23766 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:27.409138 23765 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3c878cfc75c445c97d5a171e53cde86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3c878cfc75c445c97d5a171e53cde86" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43675 } } }
I20260620 15:05:27.409610 23765 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:27.416922 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:27.399709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:39501 (local address 127.0.0.1:43675)
0620 15:05:27.399897 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:27.399904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:27.400436 (+ 532us) server_negotiation.cc:407] Connection header received
0620 15:05:27.400631 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:27.400636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:27.400718 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:27.400829 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:27.402374 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:27.403190 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:27.404722 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:27.405431 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:27.405621 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:27.408974 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:27.408981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:27.408985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:27.409028 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:27.413811 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:27.414711 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:27.414717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:27.414721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:27.414893 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:27.415349 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:27.415354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:27.415358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:27.415667 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:27.415975 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:27.416334 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 15:05:27.416511 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:05:27.422816 23772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:27.430027 23772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:27.439734 23772 catalog_manager.cc:1347] Generated new cluster ID: f312dfd821424d5ea957decd3b50174c
I20260620 15:05:27.439863 23772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:27.457548 23772 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:27.459467 23772 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:27.468171 23772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3c878cfc75c445c97d5a171e53cde86: Generated new TSK 0
I20260620 15:05:27.469270 23772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:27.479480 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23687
I20260620 15:05:27.480798 23749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:27.573681 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23687
W20260620 15:05:27.587983 23672 connection.cc:535] client connection to 127.0.0.1:43675 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43675): Connection reset by peer (error 104)
2026-06-20T15:05:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:27 dist-test-slave-n5vv krb5kdc[23786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:27 dist-test-slave-n5vv krb5kdc[23786](info): set up 2 sockets
Jun 20 15:05:27 dist-test-slave-n5vv krb5kdc[23786](info): commencing operation
krb5kdc: starting...
W20260620 15:05:28.689893 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:28 dist-test-slave-n5vv krb5kdc[23786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:28.952430 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41409
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:29.285881 23799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:29.286171 23799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:29.286360 23799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:29.293744 23799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:29.293857 23799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:29.293967 23799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:29.294078 23799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:29.294162 23799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:29.304133 23799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41409
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:29.305423 23799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:29.306763 23799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:29.318974 23804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:29.318904 23805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:29.318933 23807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:29.319785 23799 server_base.cc:1034] running on GCE node
I20260620 15:05:29.320582 23799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:29.322134 23799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:29.323360 23799 hybrid_clock.cc:648] HybridClock initialized: now 1781967929323296 us; error 31 us; skew 500 ppm
I20260620 15:05:29.323733 23799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:29 dist-test-slave-n5vv krb5kdc[23786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:29.326956 23799 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:29.328536 23799 webserver.cc:460] Webserver started at http://127.0.0.1:36899/ using document root <none> and password file <none>
I20260620 15:05:29.329223 23799 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:29.329360 23799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:29.329736 23799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:29.332724 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "a012d3babc1f4659af0365a760d3641d"
format_stamp: "Formatted at 2026-06-20 15:05:29 on dist-test-slave-n5vv"
server_key: "ce4410fd074bb2b09d98158ac7f47f22"
server_key_iv: "43f856014114c2901d04002df4578750"
server_key_version: "encryptionkey@0"
I20260620 15:05:29.333565 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "a012d3babc1f4659af0365a760d3641d"
format_stamp: "Formatted at 2026-06-20 15:05:29 on dist-test-slave-n5vv"
server_key: "ce4410fd074bb2b09d98158ac7f47f22"
server_key_iv: "43f856014114c2901d04002df4578750"
server_key_version: "encryptionkey@0"
I20260620 15:05:29.338718 23799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:05:29.342119 23814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:29.343381 23799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:29.343557 23799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "a012d3babc1f4659af0365a760d3641d"
format_stamp: "Formatted at 2026-06-20 15:05:29 on dist-test-slave-n5vv"
server_key: "ce4410fd074bb2b09d98158ac7f47f22"
server_key_iv: "43f856014114c2901d04002df4578750"
server_key_version: "encryptionkey@0"
I20260620 15:05:29.343752 23799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:29.359906 23799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:29.364125 23799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:29.364449 23799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:29.383123 23799 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41409
I20260620 15:05:29.383167 23865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41409 every 8 connection(s)
I20260620 15:05:29.384672 23799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:29.387467 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23799
I20260620 15:05:29.387642 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:29.388073 2328 external_mini_cluster.cc:1428] Setting key e46e3ad72d61989ab7b23fa0edde5508
I20260620 15:05:29.388751 23866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:29.402740 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:29.389972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:51393 (local address 127.0.0.1:41409)
0620 15:05:29.390758 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:29.390774 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:29.390824 (+ 50us) server_negotiation.cc:407] Connection header received
0620 15:05:29.392488 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:29.392516 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:29.393066 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:29.393531 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:29.394054 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:29.396092 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:29.397543 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:29.398561 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:29.398766 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:29.399932 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:29.399946 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:29.399964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:29.400689 (+ 725us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:29.400788 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:29.400826 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:29.400995 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:29.401227 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 15:05:29.401418 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":304,"threads_started":1}
I20260620 15:05:29.403821 23866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d: Bootstrap starting.
I20260620 15:05:29.406432 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:29.407141 23866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:29.408206 23866 log.cc:824] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:29.411391 23866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d: No bootstrap required, opened a new log
I20260620 15:05:29.417665 23869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:29.411038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:41529 (local address 127.0.0.1:41409)
0620 15:05:29.411205 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:29.411211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:29.411607 (+ 396us) server_negotiation.cc:407] Connection header received
0620 15:05:29.411769 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:29.411774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:29.411862 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:29.411990 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:29.412428 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:29.413628 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:29.414881 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:29.415589 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:29.415747 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:29.416815 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:29.416822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:29.416826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:29.416846 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:29.416908 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:29.416928 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:29.417051 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:29.417207 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 15:05:29.417267 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:05:29.420459 23866 raft_consensus.cc:348] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012d3babc1f4659af0365a760d3641d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260620 15:05:29.420856 23866 raft_consensus.cc:374] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:29.420970 23866 raft_consensus.cc:729] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a012d3babc1f4659af0365a760d3641d, State: Initialized, Role: FOLLOWER
I20260620 15:05:29.421646 23866 consensus_queue.cc:260] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012d3babc1f4659af0365a760d3641d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260620 15:05:29.421897 23866 raft_consensus.cc:388] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:29.422051 23866 raft_consensus.cc:482] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:29.422232 23866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:29.424389 23866 raft_consensus.cc:504] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012d3babc1f4659af0365a760d3641d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260620 15:05:29.424914 23866 leader_election.cc:302] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a012d3babc1f4659af0365a760d3641d; no voters:
I20260620 15:05:29.425577 23866 leader_election.cc:288] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:29.425702 23876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:29.427582 23866 sys_catalog.cc:564] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:29.427999 23876 raft_consensus.cc:686] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [term 1 LEADER]: Becoming Leader. State: Replica: a012d3babc1f4659af0365a760d3641d, State: Running, Role: LEADER
I20260620 15:05:29.428507 23876 consensus_queue.cc:237] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012d3babc1f4659af0365a760d3641d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } }
I20260620 15:05:29.432010 23878 sys_catalog.cc:455] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a012d3babc1f4659af0365a760d3641d. Latest consensus state: current_term: 1 leader_uuid: "a012d3babc1f4659af0365a760d3641d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012d3babc1f4659af0365a760d3641d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } } }
I20260620 15:05:29.432413 23878 sys_catalog.cc:458] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:29.431988 23877 sys_catalog.cc:455] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a012d3babc1f4659af0365a760d3641d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a012d3babc1f4659af0365a760d3641d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41409 } } }
I20260620 15:05:29.432864 23877 sys_catalog.cc:458] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:29.435248 23887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:29.441174 23887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:29.450286 23887 catalog_manager.cc:1347] Generated new cluster ID: c339ca69d1574e6a898a937908be2098
I20260620 15:05:29.450420 23887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:29.460999 23887 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:29.462424 23887 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:29.475039 23887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a012d3babc1f4659af0365a760d3641d: Generated new TSK 0
I20260620 15:05:29.476167 23887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:29.493161 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23799
W20260620 15:05:29.494560 23861 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.44:51393
W20260620 15:05:29.495350 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23799: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:29.495519 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23799
W20260620 15:05:29.512363 23784 connection.cc:535] client connection to 127.0.0.1:41409 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41409): Connection reset by peer (error 104)
2026-06-20T15:05:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1924 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:29 dist-test-slave-n5vv krb5kdc[23897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:29 dist-test-slave-n5vv krb5kdc[23897](info): set up 2 sockets
Jun 20 15:05:29 dist-test-slave-n5vv krb5kdc[23897](info): commencing operation
krb5kdc: starting...
W20260620 15:05:30.638139 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:30 dist-test-slave-n5vv krb5kdc[23897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 15:05:30.901538 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:31.237581 23910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:31.237916 23910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:31.238015 23910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:31.245277 23910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:31.245399 23910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:31.245451 23910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:31.245579 23910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:31.245630 23910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:31.262120 23910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:31.263967 23910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:31.266021 23910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:31.288054 23918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:31.287966 23916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:31.288203 23915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:31.289211 23910 server_base.cc:1034] running on GCE node
I20260620 15:05:31.290158 23910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:31.291769 23910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:31.292989 23910 hybrid_clock.cc:648] HybridClock initialized: now 1781967931292918 us; error 65 us; skew 500 ppm
I20260620 15:05:31.293388 23910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:31 dist-test-slave-n5vv krb5kdc[23897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:31.297117 23910 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 15:05:31.299155 23910 webserver.cc:460] Webserver started at http://127.0.0.1:41347/ using document root <none> and password file <none>
I20260620 15:05:31.300096 23910 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:31.300242 23910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:31.300622 23910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:31.304173 23910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "4802c5860ba74e0d8dae8e3b87e44617"
format_stamp: "Formatted at 2026-06-20 15:05:31 on dist-test-slave-n5vv"
server_key: "0e02d63c81e0e4167d1406484dcadd15"
server_key_iv: "2fd8909656244336a41e6dddf496e273"
server_key_version: "encryptionkey@0"
I20260620 15:05:31.305040 23910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "4802c5860ba74e0d8dae8e3b87e44617"
format_stamp: "Formatted at 2026-06-20 15:05:31 on dist-test-slave-n5vv"
server_key: "0e02d63c81e0e4167d1406484dcadd15"
server_key_iv: "2fd8909656244336a41e6dddf496e273"
server_key_version: "encryptionkey@0"
I20260620 15:05:31.310818 23910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260620 15:05:31.314584 23925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:31.316011 23910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:05:31.316218 23910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "4802c5860ba74e0d8dae8e3b87e44617"
format_stamp: "Formatted at 2026-06-20 15:05:31 on dist-test-slave-n5vv"
server_key: "0e02d63c81e0e4167d1406484dcadd15"
server_key_iv: "2fd8909656244336a41e6dddf496e273"
server_key_version: "encryptionkey@0"
I20260620 15:05:31.316411 23910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:31.322293 23910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:31.326277 23910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:31.326581 23910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:31.346109 23910 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36739
I20260620 15:05:31.346160 23976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36739 every 8 connection(s)
I20260620 15:05:31.347604 23910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:31.351470 23977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:31.356920 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 23910
I20260620 15:05:31.357086 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:31.357622 2328 external_mini_cluster.cc:1428] Setting key 2428fc16abcace3c573e2c6267e0f73f
I20260620 15:05:31.363058 23977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617: Bootstrap starting.
I20260620 15:05:31.365808 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:31.359573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:44359 (local address 127.0.0.1:36739)
0620 15:05:31.360409 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:31.360421 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:31.360466 (+ 45us) server_negotiation.cc:407] Connection header received
0620 15:05:31.362276 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:31.362301 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:31.362652 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:31.363057 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:31.364121 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:31.364127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:31.364140 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:31.364153 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:31.364212 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:31.364232 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:31.364470 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:31.364634 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 15:05:31.364821 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":336,"threads_started":1}
I20260620 15:05:31.366832 23977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:31.368325 23977 log.cc:824] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:31.369763 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:31.371333 23977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617: No bootstrap required, opened a new log
I20260620 15:05:31.376984 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:31.374440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.44:39675 (local address 127.0.0.1:36739)
0620 15:05:31.374669 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:31.374676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:31.374990 (+ 314us) server_negotiation.cc:407] Connection header received
0620 15:05:31.375190 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:31.375195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:31.375251 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:31.375372 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:31.376327 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:31.376332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:31.376335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:31.376352 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:31.376397 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:31.376415 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:31.376550 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:31.376640 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 15:05:31.376710 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 15:05:31.380319 23977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4802c5860ba74e0d8dae8e3b87e44617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36739 } }
I20260620 15:05:31.380694 23977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:31.380847 23977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4802c5860ba74e0d8dae8e3b87e44617, State: Initialized, Role: FOLLOWER
I20260620 15:05:31.381464 23977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4802c5860ba74e0d8dae8e3b87e44617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36739 } }
I20260620 15:05:31.381812 23977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:31.381971 23977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:31.382171 23977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:31.384868 23977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4802c5860ba74e0d8dae8e3b87e44617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36739 } }
I20260620 15:05:31.385406 23977 leader_election.cc:302] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4802c5860ba74e0d8dae8e3b87e44617; no voters:
I20260620 15:05:31.386065 23977 leader_election.cc:288] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:31.386320 23987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:31.387702 23987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [term 1 LEADER]: Becoming Leader. State: Replica: 4802c5860ba74e0d8dae8e3b87e44617, State: Running, Role: LEADER
I20260620 15:05:31.388373 23977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:31.388238 23987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4802c5860ba74e0d8dae8e3b87e44617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36739 } }
I20260620 15:05:31.391222 23989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4802c5860ba74e0d8dae8e3b87e44617. Latest consensus state: current_term: 1 leader_uuid: "4802c5860ba74e0d8dae8e3b87e44617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4802c5860ba74e0d8dae8e3b87e44617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36739 } } }
I20260620 15:05:31.391559 23988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4802c5860ba74e0d8dae8e3b87e44617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4802c5860ba74e0d8dae8e3b87e44617" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36739 } } }
I20260620 15:05:31.391835 23989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:31.391858 23988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:31.394115 23995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:31.398640 23995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:31.407260 23995 catalog_manager.cc:1347] Generated new cluster ID: 3d63254316d448f9b22cd9921aedf8e5
I20260620 15:05:31.407392 23995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:31.428485 23995 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:31.428965 23995 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:31.439530 23995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4802c5860ba74e0d8dae8e3b87e44617: Generated new TSK 0
I20260620 15:05:31.440402 23995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:31.447234 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23910
W20260620 15:05:31.448477 23972 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.44:44359
W20260620 15:05:31.449280 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 23910: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:31.449438 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 23910
2026-06-20T15:05:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:31 dist-test-slave-n5vv krb5kdc[24008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:31 dist-test-slave-n5vv krb5kdc[24008](info): set up 2 sockets
Jun 20 15:05:31 dist-test-slave-n5vv krb5kdc[24008](info): commencing operation
krb5kdc: starting...
W20260620 15:05:32.573884 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:32 dist-test-slave-n5vv krb5kdc[24008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:32.829569 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42485
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:41203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:42485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:33.160530 24021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:33.160861 24021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:33.161029 24021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:33.168344 24021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:33.168493 24021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:33.168601 24021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:33.168709 24021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:33.168804 24021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:33.178793 24021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:41203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:42485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:42485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:33.180035 24021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:33.181380 24021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:33.193234 24029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:33.193197 24027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:33.193641 24026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:33.194291 24021 server_base.cc:1034] running on GCE node
I20260620 15:05:33.195084 24021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:33.196523 24021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:33.197758 24021 hybrid_clock.cc:648] HybridClock initialized: now 1781967933197709 us; error 46 us; skew 500 ppm
I20260620 15:05:33.198118 24021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:33 dist-test-slave-n5vv krb5kdc[24008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967933, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:33.201570 24021 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:33.203141 24021 webserver.cc:460] Webserver started at http://127.2.70.62:42851/ using document root <none> and password file <none>
I20260620 15:05:33.203841 24021 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:33.203987 24021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:33.204295 24021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:33.207294 24021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "a227940a8c984dcba2e3a85690f5cd88"
format_stamp: "Formatted at 2026-06-20 15:05:33 on dist-test-slave-n5vv"
server_key: "b11cfea818d6a91684dac3a71b1a71ea"
server_key_iv: "bc98b26a4bfa68473141602e456e889c"
server_key_version: "encryptionkey@0"
I20260620 15:05:33.208047 24021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "a227940a8c984dcba2e3a85690f5cd88"
format_stamp: "Formatted at 2026-06-20 15:05:33 on dist-test-slave-n5vv"
server_key: "b11cfea818d6a91684dac3a71b1a71ea"
server_key_iv: "bc98b26a4bfa68473141602e456e889c"
server_key_version: "encryptionkey@0"
I20260620 15:05:33.213162 24021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260620 15:05:33.216675 24036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:33.217986 24021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:05:33.218204 24021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "a227940a8c984dcba2e3a85690f5cd88"
format_stamp: "Formatted at 2026-06-20 15:05:33 on dist-test-slave-n5vv"
server_key: "b11cfea818d6a91684dac3a71b1a71ea"
server_key_iv: "bc98b26a4bfa68473141602e456e889c"
server_key_version: "encryptionkey@0"
I20260620 15:05:33.218421 24021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:33.237557 24021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:33.242007 24021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:33.242307 24021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:33.260974 24021 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:42485
I20260620 15:05:33.261010 24087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:42485 every 8 connection(s)
I20260620 15:05:33.262375 24021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:33.264573 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24021
I20260620 15:05:33.264775 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:33.265331 2328 external_mini_cluster.cc:1428] Setting key 9b36d48232fc833caef0e98d31305bc0
I20260620 15:05:33.266959 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 20 15:05:33 dist-test-slave-n5vv krb5kdc[24008](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:33.280737 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88: Bootstrap starting.
I20260620 15:05:33.284006 24088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:33.285353 24088 log.cc:824] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:33.288503 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88: No bootstrap required, opened a new log
I20260620 15:05:33.288645 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:33.267214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41449 (local address 127.2.70.62:42485)
0620 15:05:33.267999 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:33.268011 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:33.268054 (+ 43us) server_negotiation.cc:407] Connection header received
0620 15:05:33.269661 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:33.269690 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:33.270074 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:33.270393 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:33.272143 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:33.273558 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:33.275326 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:33.276021 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:33.276144 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:33.280931 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:33.280940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:33.280953 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:33.280996 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:33.284016 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:33.285001 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:33.285008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:33.285014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:33.285176 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:33.285729 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:33.285733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:33.285735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:33.286300 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:33.286533 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:33.287251 (+ 718us) server_negotiation.cc:299] Negotiation successful
0620 15:05:33.287592 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":344,"threads_started":1}
I20260620 15:05:33.293030 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:33.297000 24088 raft_consensus.cc:348] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227940a8c984dcba2e3a85690f5cd88" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42485 } }
I20260620 15:05:33.297495 24088 raft_consensus.cc:374] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:33.297683 24088 raft_consensus.cc:729] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a227940a8c984dcba2e3a85690f5cd88, State: Initialized, Role: FOLLOWER
I20260620 15:05:33.298365 24088 consensus_queue.cc:260] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227940a8c984dcba2e3a85690f5cd88" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42485 } }
I20260620 15:05:33.298744 24088 raft_consensus.cc:388] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:33.298898 24088 raft_consensus.cc:482] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:33.299091 24088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:33.302115 24088 raft_consensus.cc:504] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227940a8c984dcba2e3a85690f5cd88" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42485 } }
I20260620 15:05:33.302707 24088 leader_election.cc:302] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a227940a8c984dcba2e3a85690f5cd88; no voters:
I20260620 15:05:33.303397 24088 leader_election.cc:288] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:33.303534 24098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:33.304939 24098 raft_consensus.cc:686] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [term 1 LEADER]: Becoming Leader. State: Replica: a227940a8c984dcba2e3a85690f5cd88, State: Running, Role: LEADER
I20260620 15:05:33.305572 24098 consensus_queue.cc:237] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227940a8c984dcba2e3a85690f5cd88" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42485 } }
I20260620 15:05:33.306129 24088 sys_catalog.cc:564] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:33.309545 24100 sys_catalog.cc:455] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a227940a8c984dcba2e3a85690f5cd88. Latest consensus state: current_term: 1 leader_uuid: "a227940a8c984dcba2e3a85690f5cd88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227940a8c984dcba2e3a85690f5cd88" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42485 } } }
I20260620 15:05:33.310027 24100 sys_catalog.cc:458] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:33.314632 24099 sys_catalog.cc:455] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a227940a8c984dcba2e3a85690f5cd88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a227940a8c984dcba2e3a85690f5cd88" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 42485 } } }
I20260620 15:05:33.315641 24099 sys_catalog.cc:458] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:33.317073 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:33.297358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51665 (local address 127.2.70.62:42485)
0620 15:05:33.297583 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:33.297591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:33.298000 (+ 409us) server_negotiation.cc:407] Connection header received
0620 15:05:33.298260 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:33.298264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:33.298370 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:33.298478 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:33.300383 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:33.301201 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:33.303034 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:33.303793 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:33.303952 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:33.306813 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:33.306822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:33.306829 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:33.306868 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:33.311398 (+ 4530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:33.312741 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:33.312750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:33.312756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:33.312998 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:33.313768 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:33.313776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:33.313782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:33.314310 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:33.314693 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:33.316287 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0620 15:05:33.316537 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:05:33.319010 24104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:33.324061 24104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:33.333585 24104 catalog_manager.cc:1347] Generated new cluster ID: efbed60b99214be7ad4734926570e59c
I20260620 15:05:33.333715 24104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:33.363873 24104 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:33.365250 24104 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:33.374708 24104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a227940a8c984dcba2e3a85690f5cd88: Generated new TSK 0
I20260620 15:05:33.375779 24104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:33.388800 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24021
I20260620 15:05:33.390324 24083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:33.484287 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24021
W20260620 15:05:33.501854 24006 connection.cc:535] client connection to 127.2.70.62:42485 recv error: Network error: failed to read from TLS socket (remote: 127.2.70.62:42485): Connection reset by peer (error 104)
2026-06-20T15:05:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2034 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:33 dist-test-slave-n5vv krb5kdc[24120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:33 dist-test-slave-n5vv krb5kdc[24120](info): set up 2 sockets
Jun 20 15:05:33 dist-test-slave-n5vv krb5kdc[24120](info): commencing operation
krb5kdc: starting...
W20260620 15:05:34.638047 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:34 dist-test-slave-n5vv krb5kdc[24120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:34.892545 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:40903
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:33963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:40903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:35.228013 24133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:35.228358 24133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:35.228564 24133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:35.235960 24133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:35.236073 24133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:35.236212 24133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:35.236317 24133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:35.236402 24133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:35.246551 24133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:33963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:40903
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:40903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:35.247887 24133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:35.249169 24133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:35.261104 24139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:35.261166 24141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:35.261791 24133 server_base.cc:1034] running on GCE node
W20260620 15:05:35.261889 24138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:35.262717 24133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:35.264096 24133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:35.265388 24133 hybrid_clock.cc:648] HybridClock initialized: now 1781967935265324 us; error 52 us; skew 500 ppm
I20260620 15:05:35.265764 24133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:35 dist-test-slave-n5vv krb5kdc[24120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967935, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:35.268898 24133 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:35.270463 24133 webserver.cc:460] Webserver started at http://127.2.70.62:45043/ using document root <none> and password file <none>
I20260620 15:05:35.271153 24133 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:35.271257 24133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:35.271489 24133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:35.274446 24133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "ac33265423ea475ebe97e67b7ca52a97"
format_stamp: "Formatted at 2026-06-20 15:05:35 on dist-test-slave-n5vv"
server_key: "9625a6214fbfc11df39ff0ee02664b51"
server_key_iv: "7b3d9a6c73aab491c111d0bdc044e6bb"
server_key_version: "encryptionkey@0"
I20260620 15:05:35.275158 24133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "ac33265423ea475ebe97e67b7ca52a97"
format_stamp: "Formatted at 2026-06-20 15:05:35 on dist-test-slave-n5vv"
server_key: "9625a6214fbfc11df39ff0ee02664b51"
server_key_iv: "7b3d9a6c73aab491c111d0bdc044e6bb"
server_key_version: "encryptionkey@0"
I20260620 15:05:35.280326 24133 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:05:35.283902 24148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:35.285243 24133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260620 15:05:35.285424 24133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "ac33265423ea475ebe97e67b7ca52a97"
format_stamp: "Formatted at 2026-06-20 15:05:35 on dist-test-slave-n5vv"
server_key: "9625a6214fbfc11df39ff0ee02664b51"
server_key_iv: "7b3d9a6c73aab491c111d0bdc044e6bb"
server_key_version: "encryptionkey@0"
I20260620 15:05:35.285720 24133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:35.301821 24133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:35.306264 24133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:35.306615 24133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:35.325529 24133 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:40903
I20260620 15:05:35.325541 24199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:40903 every 8 connection(s)
I20260620 15:05:35.326920 24133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:35.327196 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24133
I20260620 15:05:35.327368 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:35.327854 2328 external_mini_cluster.cc:1428] Setting key bc0f8c0b6595eb37d9b5dac4284c617b
I20260620 15:05:35.331719 24200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:35.341231 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:35.329853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57309 (local address 127.2.70.62:40903)
0620 15:05:35.330710 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:35.330722 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:35.330764 (+ 42us) server_negotiation.cc:407] Connection header received
0620 15:05:35.332533 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:35.332555 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:35.332935 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:35.333333 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:35.333880 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:35.335232 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:35.336618 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:35.337323 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:35.337542 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:35.338874 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:35.338882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:35.338895 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:35.339424 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:35.339505 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:35.339529 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:35.339725 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:35.340021 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 15:05:35.340184 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":443,"threads_started":1}
I20260620 15:05:35.344035 24200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97: Bootstrap starting.
I20260620 15:05:35.345403 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:35.347489 24200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:35.348724 24200 log.cc:824] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:35.352313 24200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97: No bootstrap required, opened a new log
I20260620 15:05:35.356941 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:35.349817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50605 (local address 127.2.70.62:40903)
0620 15:05:35.350058 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:35.350068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:35.350622 (+ 554us) server_negotiation.cc:407] Connection header received
0620 15:05:35.350758 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:35.350764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:35.350841 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:35.350961 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:35.351445 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:35.352658 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:35.354047 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:35.354838 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:35.354987 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:35.356054 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:35.356065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:35.356071 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:35.356094 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:35.356146 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:35.356169 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:35.356303 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:35.356463 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 15:05:35.356520 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 15:05:35.361866 24200 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac33265423ea475ebe97e67b7ca52a97" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40903 } }
I20260620 15:05:35.362283 24200 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:35.362458 24200 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac33265423ea475ebe97e67b7ca52a97, State: Initialized, Role: FOLLOWER
I20260620 15:05:35.363032 24200 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac33265423ea475ebe97e67b7ca52a97" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40903 } }
I20260620 15:05:35.363286 24200 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:35.363456 24200 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:35.363633 24200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:35.365960 24200 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac33265423ea475ebe97e67b7ca52a97" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40903 } }
I20260620 15:05:35.366503 24200 leader_election.cc:302] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac33265423ea475ebe97e67b7ca52a97; no voters:
I20260620 15:05:35.367091 24200 leader_election.cc:288] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:35.367324 24210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:35.368103 24210 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [term 1 LEADER]: Becoming Leader. State: Replica: ac33265423ea475ebe97e67b7ca52a97, State: Running, Role: LEADER
I20260620 15:05:35.368635 24210 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac33265423ea475ebe97e67b7ca52a97" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40903 } }
I20260620 15:05:35.369261 24200 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:35.371490 24211 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac33265423ea475ebe97e67b7ca52a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac33265423ea475ebe97e67b7ca52a97" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40903 } } }
I20260620 15:05:35.371804 24211 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:35.371974 24212 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac33265423ea475ebe97e67b7ca52a97. Latest consensus state: current_term: 1 leader_uuid: "ac33265423ea475ebe97e67b7ca52a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac33265423ea475ebe97e67b7ca52a97" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 40903 } } }
I20260620 15:05:35.372680 24218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:35.373618 24212 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:35.381052 24218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:35.390674 24218 catalog_manager.cc:1347] Generated new cluster ID: 5bd5d393b48d4fa2b360bc810e30d8c6
I20260620 15:05:35.390827 24218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:35.410022 24218 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:35.411293 24218 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:35.417604 24218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac33265423ea475ebe97e67b7ca52a97: Generated new TSK 0
I20260620 15:05:35.418444 24218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:35.434280 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24133
W20260620 15:05:35.436017 24195 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57309
W20260620 15:05:35.436827 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24133: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:35.436965 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24133
W20260620 15:05:35.451130 24118 connection.cc:535] client connection to 127.2.70.62:40903 recv error: Network error: failed to read from TLS socket (remote: 127.2.70.62:40903): Connection reset by peer (error 104)
2026-06-20T15:05:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:35 dist-test-slave-n5vv krb5kdc[24231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:35 dist-test-slave-n5vv krb5kdc[24231](info): set up 2 sockets
Jun 20 15:05:35 dist-test-slave-n5vv krb5kdc[24231](info): commencing operation
krb5kdc: starting...
W20260620 15:05:36.549984 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:36 dist-test-slave-n5vv krb5kdc[24231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:36.804729 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:36001
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:36001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:37.140616 24244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:37.140949 24244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:37.141117 24244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:37.148352 24244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:37.148504 24244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:37.148607 24244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:37.148715 24244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:37.148794 24244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:37.159482 24244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:36001
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:36001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:37.160787 24244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:37.162194 24244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:37.174455 24249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:37.174688 24250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:37.174634 24252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:37.175720 24244 server_base.cc:1034] running on GCE node
I20260620 15:05:37.176587 24244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:37.178336 24244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:37.179596 24244 hybrid_clock.cc:648] HybridClock initialized: now 1781967937179550 us; error 43 us; skew 500 ppm
I20260620 15:05:37.180007 24244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:37 dist-test-slave-n5vv krb5kdc[24231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967937, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:37.187266 24244 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:37.189000 24244 webserver.cc:460] Webserver started at http://127.2.70.62:34357/ using document root <none> and password file <none>
I20260620 15:05:37.189821 24244 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:37.189966 24244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:37.190294 24244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:37.193428 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "626b32ac577244dfb0fe95fe5b5b3625"
format_stamp: "Formatted at 2026-06-20 15:05:37 on dist-test-slave-n5vv"
server_key: "0154316d81a01b1a6c39967c91149375"
server_key_iv: "3f0e55e86344e07c991b1df86adb7b56"
server_key_version: "encryptionkey@0"
I20260620 15:05:37.194306 24244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "626b32ac577244dfb0fe95fe5b5b3625"
format_stamp: "Formatted at 2026-06-20 15:05:37 on dist-test-slave-n5vv"
server_key: "0154316d81a01b1a6c39967c91149375"
server_key_iv: "3f0e55e86344e07c991b1df86adb7b56"
server_key_version: "encryptionkey@0"
I20260620 15:05:37.199985 24244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:05:37.203879 24259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:37.205686 24244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:05:37.205896 24244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "626b32ac577244dfb0fe95fe5b5b3625"
format_stamp: "Formatted at 2026-06-20 15:05:37 on dist-test-slave-n5vv"
server_key: "0154316d81a01b1a6c39967c91149375"
server_key_iv: "3f0e55e86344e07c991b1df86adb7b56"
server_key_version: "encryptionkey@0"
I20260620 15:05:37.206120 24244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:37.212100 24244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:37.216697 24244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:37.217033 24244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:37.236982 24244 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:36001
I20260620 15:05:37.237025 24310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:36001 every 8 connection(s)
I20260620 15:05:37.238479 24244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:37.238790 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24244
I20260620 15:05:37.238976 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:37.239588 2328 external_mini_cluster.cc:1428] Setting key 2b7e1b47ab8a31304613bc56bb3eb95f
I20260620 15:05:37.243222 24311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:37.247417 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:37.241455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51679 (local address 127.2.70.62:36001)
0620 15:05:37.242255 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:37.242271 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:37.242317 (+ 46us) server_negotiation.cc:407] Connection header received
0620 15:05:37.244007 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:37.244031 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:37.244378 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:37.244757 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:37.245820 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:37.245829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:37.245844 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:37.245857 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:37.245931 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:37.245948 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:37.246195 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:37.246349 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 15:05:37.246524 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":316,"threads_started":1}
I20260620 15:05:37.251123 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:37.254227 24311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625: Bootstrap starting.
I20260620 15:05:37.257800 24311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:37.258069 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:37.255439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40407 (local address 127.2.70.62:36001)
0620 15:05:37.255618 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:37.255623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:37.256068 (+ 445us) server_negotiation.cc:407] Connection header received
0620 15:05:37.256260 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:37.256270 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:37.256320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:37.256438 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:37.257362 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:37.257374 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:37.257378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:37.257393 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:37.257430 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:37.257444 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:37.257602 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:37.257751 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 15:05:37.257814 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 15:05:37.259016 24311 log.cc:824] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:37.261732 24311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625: No bootstrap required, opened a new log
I20260620 15:05:37.268788 24311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626b32ac577244dfb0fe95fe5b5b3625" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36001 } }
I20260620 15:05:37.269176 24311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:37.269321 24311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626b32ac577244dfb0fe95fe5b5b3625, State: Initialized, Role: FOLLOWER
I20260620 15:05:37.270006 24311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626b32ac577244dfb0fe95fe5b5b3625" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36001 } }
I20260620 15:05:37.270257 24311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:37.270393 24311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:37.270578 24311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:37.272694 24311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626b32ac577244dfb0fe95fe5b5b3625" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36001 } }
I20260620 15:05:37.273200 24311 leader_election.cc:302] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 626b32ac577244dfb0fe95fe5b5b3625; no voters:
I20260620 15:05:37.273849 24311 leader_election.cc:288] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:37.273976 24321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:37.274607 24321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [term 1 LEADER]: Becoming Leader. State: Replica: 626b32ac577244dfb0fe95fe5b5b3625, State: Running, Role: LEADER
I20260620 15:05:37.275390 24321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626b32ac577244dfb0fe95fe5b5b3625" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36001 } }
I20260620 15:05:37.277082 24311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:37.278389 24323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "626b32ac577244dfb0fe95fe5b5b3625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626b32ac577244dfb0fe95fe5b5b3625" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36001 } } }
I20260620 15:05:37.278337 24322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 626b32ac577244dfb0fe95fe5b5b3625. Latest consensus state: current_term: 1 leader_uuid: "626b32ac577244dfb0fe95fe5b5b3625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "626b32ac577244dfb0fe95fe5b5b3625" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 36001 } } }
I20260620 15:05:37.278776 24323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:37.278831 24322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:37.286628 24326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:37.291355 24326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:37.299567 24326 catalog_manager.cc:1347] Generated new cluster ID: 8a04833e45064478b031c93601fac402
I20260620 15:05:37.299686 24326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:37.308290 24326 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:37.308669 24326 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:37.315263 24326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 626b32ac577244dfb0fe95fe5b5b3625: Generated new TSK 0
I20260620 15:05:37.315999 24326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:37.320974 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24244
W20260620 15:05:37.322330 24306 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51679
W20260620 15:05:37.322966 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24244: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:37.323117 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24244
2026-06-20T15:05:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1887 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:37 dist-test-slave-n5vv krb5kdc[24342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:37 dist-test-slave-n5vv krb5kdc[24342](info): set up 2 sockets
Jun 20 15:05:37 dist-test-slave-n5vv krb5kdc[24342](info): commencing operation
krb5kdc: starting...
W20260620 15:05:38.426041 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:38 dist-test-slave-n5vv krb5kdc[24342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:38.693116 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45731
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:45731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:39.025895 24355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:39.026245 24355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:39.026417 24355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:39.035072 24355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:39.035230 24355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:39.035356 24355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:39.035473 24355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:39.035559 24355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:39.047066 24355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:45731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:45731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:39.048410 24355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:39.049809 24355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:39.061956 24360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:39.062089 24363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:39.062479 24361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:39.063262 24355 server_base.cc:1034] running on GCE node
I20260620 15:05:39.064111 24355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:39.065667 24355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:39.066895 24355 hybrid_clock.cc:648] HybridClock initialized: now 1781967939066858 us; error 34 us; skew 500 ppm
I20260620 15:05:39.067320 24355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:39 dist-test-slave-n5vv krb5kdc[24342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967939, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:39.070569 24355 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:39.072181 24355 webserver.cc:460] Webserver started at http://127.2.70.62:35895/ using document root <none> and password file <none>
I20260620 15:05:39.072901 24355 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:39.073046 24355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:39.073423 24355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:39.076442 24355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "b50508834da84c959d2edd56385410ea"
format_stamp: "Formatted at 2026-06-20 15:05:39 on dist-test-slave-n5vv"
server_key: "6183a0187c0baacc14522b4ec199b1dc"
server_key_iv: "cadf6009384997fa68340324e5f21339"
server_key_version: "encryptionkey@0"
I20260620 15:05:39.077241 24355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "b50508834da84c959d2edd56385410ea"
format_stamp: "Formatted at 2026-06-20 15:05:39 on dist-test-slave-n5vv"
server_key: "6183a0187c0baacc14522b4ec199b1dc"
server_key_iv: "cadf6009384997fa68340324e5f21339"
server_key_version: "encryptionkey@0"
I20260620 15:05:39.082654 24355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:05:39.086299 24370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:39.087644 24355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:39.087841 24355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "b50508834da84c959d2edd56385410ea"
format_stamp: "Formatted at 2026-06-20 15:05:39 on dist-test-slave-n5vv"
server_key: "6183a0187c0baacc14522b4ec199b1dc"
server_key_iv: "cadf6009384997fa68340324e5f21339"
server_key_version: "encryptionkey@0"
I20260620 15:05:39.088052 24355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:39.104523 24355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:39.108727 24355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:39.109027 24355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:39.132042 24355 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:45731
I20260620 15:05:39.132073 24421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:45731 every 8 connection(s)
I20260620 15:05:39.134001 24355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:39.137929 24422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:39.138604 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24355
I20260620 15:05:39.138784 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:39.139420 2328 external_mini_cluster.cc:1428] Setting key 4ba98a32562180e63e780164ebb39bf6
I20260620 15:05:39.150532 24422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea: Bootstrap starting.
Jun 20 15:05:39 dist-test-slave-n5vv krb5kdc[24342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:39.153995 24422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:39.155012 24422 log.cc:824] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:39.158007 24422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea: No bootstrap required, opened a new log
I20260620 15:05:39.162081 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:39.141362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59371 (local address 127.2.70.62:45731)
0620 15:05:39.142215 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:39.142227 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:39.142267 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:05:39.144372 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:39.144397 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:39.144810 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:39.145226 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:39.146740 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:39.148059 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:39.149355 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:39.150115 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:39.150316 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:39.154572 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:39.154580 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:39.154594 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:39.154637 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:39.157920 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:39.158649 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:39.158656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:39.158662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:39.158822 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:39.159248 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:39.159255 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:39.159262 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:39.159804 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:39.160131 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:39.160571 (+ 440us) server_negotiation.cc:299] Negotiation successful
0620 15:05:39.160994 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":330,"threads_started":1}
I20260620 15:05:39.166033 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:39.166658 24422 raft_consensus.cc:348] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50508834da84c959d2edd56385410ea" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45731 } }
I20260620 15:05:39.167018 24422 raft_consensus.cc:374] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:39.167135 24422 raft_consensus.cc:729] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b50508834da84c959d2edd56385410ea, State: Initialized, Role: FOLLOWER
I20260620 15:05:39.167722 24422 consensus_queue.cc:260] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50508834da84c959d2edd56385410ea" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45731 } }
I20260620 15:05:39.167960 24422 raft_consensus.cc:388] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:39.168102 24422 raft_consensus.cc:482] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:39.168282 24422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:39.171226 24422 raft_consensus.cc:504] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50508834da84c959d2edd56385410ea" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45731 } }
I20260620 15:05:39.171713 24422 leader_election.cc:302] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b50508834da84c959d2edd56385410ea; no voters:
I20260620 15:05:39.172384 24422 leader_election.cc:288] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:39.172492 24432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:39.175006 24422 sys_catalog.cc:564] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:39.179085 24432 raft_consensus.cc:686] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [term 1 LEADER]: Becoming Leader. State: Replica: b50508834da84c959d2edd56385410ea, State: Running, Role: LEADER
I20260620 15:05:39.179867 24432 consensus_queue.cc:237] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50508834da84c959d2edd56385410ea" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45731 } }
I20260620 15:05:39.184477 24425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:39.170515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52931 (local address 127.2.70.62:45731)
0620 15:05:39.170709 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:39.170717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:39.171141 (+ 424us) server_negotiation.cc:407] Connection header received
0620 15:05:39.171383 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:39.171388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:39.171466 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:39.171580 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:39.173223 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:39.174065 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:39.175440 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:39.175928 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:39.176090 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:39.178656 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:39.178661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:39.178664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:39.178694 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:39.181796 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:39.182455 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:39.182460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:39.182464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:39.182592 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:39.182988 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:39.182993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:39.182997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:39.183318 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:39.183567 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:39.183891 (+ 324us) server_negotiation.cc:299] Negotiation successful
0620 15:05:39.184106 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 15:05:39.190280 24434 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b50508834da84c959d2edd56385410ea. Latest consensus state: current_term: 1 leader_uuid: "b50508834da84c959d2edd56385410ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50508834da84c959d2edd56385410ea" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45731 } } }
I20260620 15:05:39.190289 24433 sys_catalog.cc:455] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b50508834da84c959d2edd56385410ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b50508834da84c959d2edd56385410ea" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 45731 } } }
I20260620 15:05:39.196866 24433 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:39.197458 24434 sys_catalog.cc:458] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:39.199839 24442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:39.204423 24442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:39.213889 24442 catalog_manager.cc:1347] Generated new cluster ID: aec451deef8140d08964af30724017da
I20260620 15:05:39.214023 24442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:39.233920 24442 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:39.235947 24442 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:39.254837 24442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b50508834da84c959d2edd56385410ea: Generated new TSK 0
I20260620 15:05:39.256006 24442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:39.276002 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24355
I20260620 15:05:39.277818 24417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:39.379117 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24355
2026-06-20T15:05:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2057 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:39 dist-test-slave-n5vv krb5kdc[24454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:39 dist-test-slave-n5vv krb5kdc[24454](info): set up 2 sockets
Jun 20 15:05:39 dist-test-slave-n5vv krb5kdc[24454](info): commencing operation
krb5kdc: starting...
W20260620 15:05:40.494098 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:40 dist-test-slave-n5vv krb5kdc[24454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967940, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:40.755185 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:39995
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:39995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:41.084553 24467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:41.084892 24467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:41.085075 24467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:41.092381 24467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:41.092512 24467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:41.092604 24467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:41.092731 24467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:41.092805 24467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:41.103317 24467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:39995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:39995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:41.104660 24467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:41.106040 24467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:41.118268 24475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:41.118353 24472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:41.118325 24473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:41.119508 24467 server_base.cc:1034] running on GCE node
I20260620 15:05:41.120285 24467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:41.121735 24467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:41.122929 24467 hybrid_clock.cc:648] HybridClock initialized: now 1781967941122864 us; error 57 us; skew 500 ppm
I20260620 15:05:41.123216 24467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:41 dist-test-slave-n5vv krb5kdc[24454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967941, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:41.126503 24467 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:41.127967 24467 webserver.cc:460] Webserver started at http://127.2.70.62:34079/ using document root <none> and password file <none>
I20260620 15:05:41.128702 24467 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:41.128834 24467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:41.129148 24467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:41.133284 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "2782c54b7d7c43c7b1f7b111404102fc"
format_stamp: "Formatted at 2026-06-20 15:05:41 on dist-test-slave-n5vv"
server_key: "5cd5eef4f7e3b147c3bff80143e69f23"
server_key_iv: "4bade242b0fcc889c93c2cd2d804986b"
server_key_version: "encryptionkey@0"
I20260620 15:05:41.134248 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "2782c54b7d7c43c7b1f7b111404102fc"
format_stamp: "Formatted at 2026-06-20 15:05:41 on dist-test-slave-n5vv"
server_key: "5cd5eef4f7e3b147c3bff80143e69f23"
server_key_iv: "4bade242b0fcc889c93c2cd2d804986b"
server_key_version: "encryptionkey@0"
I20260620 15:05:41.139359 24467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:05:41.142845 24482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:41.144217 24467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:41.144407 24467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "2782c54b7d7c43c7b1f7b111404102fc"
format_stamp: "Formatted at 2026-06-20 15:05:41 on dist-test-slave-n5vv"
server_key: "5cd5eef4f7e3b147c3bff80143e69f23"
server_key_iv: "4bade242b0fcc889c93c2cd2d804986b"
server_key_version: "encryptionkey@0"
I20260620 15:05:41.144613 24467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:41.170913 24467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:41.175079 24467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:41.175381 24467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:41.195082 24467 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:39995
I20260620 15:05:41.195107 24533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:39995 every 8 connection(s)
I20260620 15:05:41.196651 24467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:41.199887 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24467
I20260620 15:05:41.200099 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:41.200558 2328 external_mini_cluster.cc:1428] Setting key 76ffc4deddc99b6de995d22b69ccb509
I20260620 15:05:41.201427 24534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:41.218459 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:41.203844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51567 (local address 127.2.70.62:39995)
0620 15:05:41.204762 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:41.204780 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:41.204841 (+ 61us) server_negotiation.cc:407] Connection header received
0620 15:05:41.207176 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:41.207210 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:41.207844 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:41.208377 (+ 533us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:41.208978 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:41.211332 (+ 2354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:41.212712 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:41.213954 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:41.214168 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:41.215394 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:41.215405 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:41.215426 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:41.216274 (+ 848us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:41.216388 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:41.216428 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:41.216555 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:41.216724 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 15:05:41.217004 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":378,"threads_started":1}
I20260620 15:05:41.218909 24534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc: Bootstrap starting.
I20260620 15:05:41.222295 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:41.222605 24534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:41.223670 24534 log.cc:824] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:41.227154 24534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc: No bootstrap required, opened a new log
I20260620 15:05:41.233181 24537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:41.227009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45833 (local address 127.2.70.62:39995)
0620 15:05:41.227166 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:41.227171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:41.227694 (+ 523us) server_negotiation.cc:407] Connection header received
0620 15:05:41.227918 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:41.227922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:41.227981 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:41.228091 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:41.228600 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:41.229332 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:41.230731 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:41.231181 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:41.231317 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:41.232448 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:41.232453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:41.232456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:41.232473 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:41.232513 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:41.232528 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:41.232637 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:41.232773 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 15:05:41.232868 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 15:05:41.236387 24534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782c54b7d7c43c7b1f7b111404102fc" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 39995 } }
I20260620 15:05:41.236894 24534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:41.237022 24534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2782c54b7d7c43c7b1f7b111404102fc, State: Initialized, Role: FOLLOWER
I20260620 15:05:41.237720 24534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782c54b7d7c43c7b1f7b111404102fc" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 39995 } }
I20260620 15:05:41.237987 24534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:41.238135 24534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:41.238332 24534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:41.240774 24534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782c54b7d7c43c7b1f7b111404102fc" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 39995 } }
I20260620 15:05:41.241379 24534 leader_election.cc:302] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2782c54b7d7c43c7b1f7b111404102fc; no voters:
I20260620 15:05:41.242043 24534 leader_election.cc:288] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:41.242146 24544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:41.243399 24544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [term 1 LEADER]: Becoming Leader. State: Replica: 2782c54b7d7c43c7b1f7b111404102fc, State: Running, Role: LEADER
I20260620 15:05:41.243891 24544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782c54b7d7c43c7b1f7b111404102fc" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 39995 } }
I20260620 15:05:41.244445 24534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:41.246640 24546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2782c54b7d7c43c7b1f7b111404102fc. Latest consensus state: current_term: 1 leader_uuid: "2782c54b7d7c43c7b1f7b111404102fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782c54b7d7c43c7b1f7b111404102fc" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 39995 } } }
I20260620 15:05:41.247201 24546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:41.246652 24545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2782c54b7d7c43c7b1f7b111404102fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2782c54b7d7c43c7b1f7b111404102fc" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 39995 } } }
I20260620 15:05:41.247931 24545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:41.248324 24551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:41.255864 24551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:41.264101 24551 catalog_manager.cc:1347] Generated new cluster ID: ad434444f4b24a8280aa36937920f279
I20260620 15:05:41.264215 24551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:41.277848 24551 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:41.279403 24551 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:41.291332 24551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2782c54b7d7c43c7b1f7b111404102fc: Generated new TSK 0
I20260620 15:05:41.292165 24551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:41.298527 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24467
W20260620 15:05:41.299701 24529 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51567
W20260620 15:05:41.300458 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24467: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:41.300611 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24467
2026-06-20T15:05:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1934 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:41 dist-test-slave-n5vv krb5kdc[24565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:41 dist-test-slave-n5vv krb5kdc[24565](info): set up 2 sockets
Jun 20 15:05:41 dist-test-slave-n5vv krb5kdc[24565](info): commencing operation
krb5kdc: starting...
W20260620 15:05:42.425953 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:42 dist-test-slave-n5vv krb5kdc[24565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:42.694990 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:37679
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:42089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:37679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:43.022152 24578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:43.022437 24578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:43.022601 24578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:43.029731 24578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:43.029837 24578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:43.029934 24578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:43.030019 24578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:43.030084 24578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:43.039873 24578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:42089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:37679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:37679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:43.041235 24578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:43.042578 24578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:43.055687 24586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:43.055760 24584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:43.055720 24583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:43.056663 24578 server_base.cc:1034] running on GCE node
I20260620 15:05:43.057438 24578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:43.058926 24578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:43.060160 24578 hybrid_clock.cc:648] HybridClock initialized: now 1781967943060123 us; error 36 us; skew 500 ppm
I20260620 15:05:43.060520 24578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:43 dist-test-slave-n5vv krb5kdc[24565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967943, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:43.066807 24578 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:43.068357 24578 webserver.cc:460] Webserver started at http://127.2.70.62:33269/ using document root <none> and password file <none>
I20260620 15:05:43.069031 24578 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:43.069173 24578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:43.069533 24578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:43.072446 24578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "3f6892e05d914c00a76202c26893774d"
format_stamp: "Formatted at 2026-06-20 15:05:43 on dist-test-slave-n5vv"
server_key: "20562542c66f248dbdf785b1a3c6821f"
server_key_iv: "24278d65fbdd30402334c9f318fa6230"
server_key_version: "encryptionkey@0"
I20260620 15:05:43.073201 24578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "3f6892e05d914c00a76202c26893774d"
format_stamp: "Formatted at 2026-06-20 15:05:43 on dist-test-slave-n5vv"
server_key: "20562542c66f248dbdf785b1a3c6821f"
server_key_iv: "24278d65fbdd30402334c9f318fa6230"
server_key_version: "encryptionkey@0"
I20260620 15:05:43.078352 24578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260620 15:05:43.081888 24593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:43.083194 24578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 15:05:43.083395 24578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "3f6892e05d914c00a76202c26893774d"
format_stamp: "Formatted at 2026-06-20 15:05:43 on dist-test-slave-n5vv"
server_key: "20562542c66f248dbdf785b1a3c6821f"
server_key_iv: "24278d65fbdd30402334c9f318fa6230"
server_key_version: "encryptionkey@0"
I20260620 15:05:43.083593 24578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:43.089290 24578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:43.093564 24578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:43.093851 24578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:43.112900 24578 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:37679
I20260620 15:05:43.112942 24644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:37679 every 8 connection(s)
I20260620 15:05:43.114355 24578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:43.118181 24645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:43.120240 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24578
I20260620 15:05:43.120446 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:43.121006 2328 external_mini_cluster.cc:1428] Setting key 0a7c0f68ec450ea797ddaf9b89eca835
I20260620 15:05:43.128768 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:43.122842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43073 (local address 127.2.70.62:37679)
0620 15:05:43.123708 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:43.123720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:43.123760 (+ 40us) server_negotiation.cc:407] Connection header received
0620 15:05:43.125317 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:43.125338 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:43.125768 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:43.126089 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:43.127249 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:43.127255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:43.127267 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:43.127279 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:43.127324 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:43.127341 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:43.127503 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:43.127648 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 15:05:43.127862 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":439,"threads_started":1}
I20260620 15:05:43.131803 24645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d: Bootstrap starting.
I20260620 15:05:43.132467 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:43.135412 24645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:43.136790 24645 log.cc:824] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:43.140676 24645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d: No bootstrap required, opened a new log
I20260620 15:05:43.141124 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:43.137684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38779 (local address 127.2.70.62:37679)
0620 15:05:43.137895 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:43.137905 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:43.138439 (+ 534us) server_negotiation.cc:407] Connection header received
0620 15:05:43.138751 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:43.138757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:43.138812 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:43.138910 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:43.140169 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:43.140174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:43.140179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 15:05:43.140199 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:43.140243 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 15:05:43.140263 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:43.140378 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:43.140632 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 15:05:43.140738 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:05:43.148056 24645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6892e05d914c00a76202c26893774d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 37679 } }
I20260620 15:05:43.148428 24645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 15:05:43.148535 24645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6892e05d914c00a76202c26893774d, State: Initialized, Role: FOLLOWER
I20260620 15:05:43.149094 24645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6892e05d914c00a76202c26893774d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 37679 } }
I20260620 15:05:43.149339 24645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 15:05:43.149490 24645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 15:05:43.149681 24645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:43.152011 24645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6892e05d914c00a76202c26893774d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 37679 } }
I20260620 15:05:43.152530 24645 leader_election.cc:302] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f6892e05d914c00a76202c26893774d; no voters:
I20260620 15:05:43.153085 24645 leader_election.cc:288] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 15:05:43.153267 24655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:43.153939 24655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [term 1 LEADER]: Becoming Leader. State: Replica: 3f6892e05d914c00a76202c26893774d, State: Running, Role: LEADER
I20260620 15:05:43.154469 24655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6892e05d914c00a76202c26893774d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 37679 } }
I20260620 15:05:43.155143 24645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:43.157465 24656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f6892e05d914c00a76202c26893774d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6892e05d914c00a76202c26893774d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 37679 } } }
I20260620 15:05:43.158036 24656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:43.157374 24657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f6892e05d914c00a76202c26893774d. Latest consensus state: current_term: 1 leader_uuid: "3f6892e05d914c00a76202c26893774d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6892e05d914c00a76202c26893774d" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 37679 } } }
I20260620 15:05:43.158993 24657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:43.159660 24664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:43.164894 24664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:43.174429 24664 catalog_manager.cc:1347] Generated new cluster ID: a8426e6b76a64f50a9982e438545a81e
I20260620 15:05:43.174566 24664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:43.188531 24664 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:43.189026 24664 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:43.198083 24664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f6892e05d914c00a76202c26893774d: Generated new TSK 0
I20260620 15:05:43.198963 24664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:43.206547 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24578
W20260620 15:05:43.207767 24640 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43073
W20260620 15:05:43.208521 2328 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260620 15:05:43.208700 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24578
2026-06-20T15:05:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1895 ms)
[----------] 18 tests from AuthTokenIssuingTest (35350 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:43 dist-test-slave-n5vv krb5kdc[24676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:43 dist-test-slave-n5vv krb5kdc[24676](info): set up 2 sockets
Jun 20 15:05:43 dist-test-slave-n5vv krb5kdc[24676](info): commencing operation
krb5kdc: starting...
W20260620 15:05:44.322213 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:44 dist-test-slave-n5vv krb5kdc[24676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:44.589475 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:33151
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:33151,127.2.70.61:45601,127.2.70.60:35615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:44.924352 24689 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:05:44.924707 24689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:44.924837 24689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:44.924970 24689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:44.932245 24689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:44.932340 24689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:44.932431 24689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:44.932516 24689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:44.932631 24689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:44.942484 24689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:33151,127.2.70.61:45601,127.2.70.60:35615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:33151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:44.943820 24689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:44.945139 24689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:44.957330 24695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:44.957420 24694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:44.957345 24697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:44.958194 24689 server_base.cc:1034] running on GCE node
I20260620 15:05:44.959035 24689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:44.960561 24689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:44.961862 24689 hybrid_clock.cc:648] HybridClock initialized: now 1781967944961775 us; error 45 us; skew 500 ppm
I20260620 15:05:44.962208 24689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:44 dist-test-slave-n5vv krb5kdc[24676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:44.965577 24689 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:44.967041 24689 webserver.cc:460] Webserver started at http://127.2.70.62:43447/ using document root <none> and password file <none>
I20260620 15:05:44.967753 24689 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:44.967897 24689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:44.968256 24689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:44.971386 24689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "394419b406004407a7ea5d39285fded4"
format_stamp: "Formatted at 2026-06-20 15:05:44 on dist-test-slave-n5vv"
server_key: "babb3042458444deee5b95fc5586df7c"
server_key_iv: "27528bc5af1e6c146fe522fdadd85af0"
server_key_version: "encryptionkey@0"
I20260620 15:05:44.972187 24689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "394419b406004407a7ea5d39285fded4"
format_stamp: "Formatted at 2026-06-20 15:05:44 on dist-test-slave-n5vv"
server_key: "babb3042458444deee5b95fc5586df7c"
server_key_iv: "27528bc5af1e6c146fe522fdadd85af0"
server_key_version: "encryptionkey@0"
I20260620 15:05:44.977463 24689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260620 15:05:44.980933 24704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:44.982419 24689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 15:05:44.982616 24689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "394419b406004407a7ea5d39285fded4"
format_stamp: "Formatted at 2026-06-20 15:05:44 on dist-test-slave-n5vv"
server_key: "babb3042458444deee5b95fc5586df7c"
server_key_iv: "27528bc5af1e6c146fe522fdadd85af0"
server_key_version: "encryptionkey@0"
I20260620 15:05:44.982851 24689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:45.012485 24689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:45.017220 24689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:45.017551 24689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:45.037498 24689 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:33151
I20260620 15:05:45.037587 24755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:33151 every 8 connection(s)
I20260620 15:05:45.038892 24689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:45.043259 24756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:45.046895 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24689
I20260620 15:05:45.047103 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:45.047947 2328 external_mini_cluster.cc:1428] Setting key 90911a686fae6ef4c471bfd67facf556
I20260620 15:05:45.052245 24756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:45 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.61@KRBTEST.COM" created.
I20260620 15:05:45.079411 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.056306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:33151 (local address 127.0.0.1:37108)
0620 15:05:45.057299 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.057351 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:45.060191 (+ 2840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:45.061265 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:45.061271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:45.061904 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:45.062244 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:45.062265 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.064515 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.064519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:45.065715 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:45.065727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.067074 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.067079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:45.067208 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:45.067883 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:45.067905 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:45.070359 (+ 2454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:45.074824 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:45.074834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:45.074843 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:45.075245 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:45.075720 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:45.075723 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:45.075725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:45.075874 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:45.076887 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:45.076893 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:45.077114 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 15:05:45.077377 (+ 263us) client_negotiation.cc:240] Negotiation successful
0620 15:05:45.077899 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":430,"threads_started":1}
I20260620 15:05:45.079445 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.056306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.2.70.62:33151)
0620 15:05:45.058338 (+ 2032us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:45.058343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:45.058387 (+ 44us) server_negotiation.cc:407] Connection header received
0620 15:05:45.060603 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:45.060614 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:45.060929 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:45.061150 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:45.062468 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.064352 (+ 1884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.065850 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.066851 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.067040 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:45.070626 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:45.070638 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:45.070657 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:45.070698 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:45.074695 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:45.075409 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:45.075420 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:45.075425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:45.075607 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:45.076101 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:45.076108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:45.076112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:45.076738 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:45.076903 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:45.077584 (+ 681us) server_negotiation.cc:299] Negotiation successful
0620 15:05:45.078042 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":192,"server-negotiator.queue_time_us":1837,"thread_start_us":1589,"threads_started":1}
I20260620 15:05:45.082873 24756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:05:45.084179 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.083730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47218)
0620 15:05:45.083915 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.084020 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 15:05:45.084465 24706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
I20260620 15:05:45.086206 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.085749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47220)
0620 15:05:45.085902 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.086006 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 15:05:45.086404 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:45601: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab.
Entry for principal kudu/127.2.70.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab.
I20260620 15:05:45.110674 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.61:45601
--webserver_interface=127.2.70.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab
--principal=kudu/127.2.70.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:33151,127.2.70.61:45601,127.2.70.60:35615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:05:45.135843 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } attempt: 1
I20260620 15:05:45.137136 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.136603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47222)
0620 15:05:45.136836 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.136948 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260620 15:05:45.138301 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.137900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47224)
0620 15:05:45.138053 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.138151 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:05:45.138517 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:45601: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
I20260620 15:05:45.206796 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } attempt: 2
I20260620 15:05:45.208069 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.207493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47240)
0620 15:05:45.207789 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.207908 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:05:45.209157 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.208746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47254)
0620 15:05:45.208903 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.209000 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:05:45.209369 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:45601: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
I20260620 15:05:45.300660 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } attempt: 3
I20260620 15:05:45.302091 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.301440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47268)
0620 15:05:45.301796 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.301911 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260620 15:05:45.303437 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.302919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47276)
0620 15:05:45.303113 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.303223 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:05:45.303669 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:45601: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
W20260620 15:05:45.446136 24763 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:05:45.446470 24763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:45.446540 24763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:45.446621 24763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:05:45.446964 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } attempt: 4
I20260620 15:05:45.448217 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.447581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47290)
0620 15:05:45.447926 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.448059 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205}
I20260620 15:05:45.449234 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.448855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47296)
0620 15:05:45.449006 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.449090 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:05:45.449429 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:45601: Network error: Client connection negotiation failed: client connection to 127.2.70.61:45601: connect: Connection refused (error 111)
W20260620 15:05:45.454892 24763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:45.454978 24763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:45.455036 24763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:45.455085 24763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:45.455139 24763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:45.465184 24763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:33151,127.2.70.61:45601,127.2.70.60:35615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab
--principal=kudu/127.2.70.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.61:45601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.70.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:45.466568 24763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:45.467846 24763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:45.480093 24768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:45.480043 24771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:45.480146 24769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:45.481010 24763 server_base.cc:1034] running on GCE node
I20260620 15:05:45.481922 24763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:45.483405 24763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:45.484678 24763 hybrid_clock.cc:648] HybridClock initialized: now 1781967945484582 us; error 75 us; skew 500 ppm
I20260620 15:05:45.485037 24763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:45 dist-test-slave-n5vv krb5kdc[24676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:45.488651 24763 init.cc:377] Logged in from keytab as kudu/127.2.70.61@KRBTEST.COM (short username kudu)
I20260620 15:05:45.490324 24763 webserver.cc:460] Webserver started at http://127.2.70.61:36739/ using document root <none> and password file <none>
I20260620 15:05:45.491048 24763 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:45.491180 24763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:45.491503 24763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:45.494649 24763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/data/instance:
uuid: "64150268c73f413a887b53d509f7d432"
format_stamp: "Formatted at 2026-06-20 15:05:45 on dist-test-slave-n5vv"
server_key: "ba532fe488c6b56dcbf200b72b9dc49c"
server_key_iv: "3c9da38cc28e564b36368f874751f8c3"
server_key_version: "encryptionkey@0"
I20260620 15:05:45.495440 24763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/wal/instance:
uuid: "64150268c73f413a887b53d509f7d432"
format_stamp: "Formatted at 2026-06-20 15:05:45 on dist-test-slave-n5vv"
server_key: "ba532fe488c6b56dcbf200b72b9dc49c"
server_key_iv: "3c9da38cc28e564b36368f874751f8c3"
server_key_version: "encryptionkey@0"
I20260620 15:05:45.500572 24763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 15:05:45.504138 24778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:45.505915 24763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 15:05:45.506178 24763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/wal
uuid: "64150268c73f413a887b53d509f7d432"
format_stamp: "Formatted at 2026-06-20 15:05:45 on dist-test-slave-n5vv"
server_key: "ba532fe488c6b56dcbf200b72b9dc49c"
server_key_iv: "3c9da38cc28e564b36368f874751f8c3"
server_key_version: "encryptionkey@0"
I20260620 15:05:45.506467 24763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:45.543838 24763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:45.548470 24763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:45.548758 24763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:45.568176 24763 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.61:45601
I20260620 15:05:45.568209 24829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.61:45601 every 8 connection(s)
I20260620 15:05:45.569595 24763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
I20260620 15:05:45.573650 24830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:45.576452 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24763
I20260620 15:05:45.576650 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-1/wal/instance
I20260620 15:05:45.577203 2328 external_mini_cluster.cc:1428] Setting key 907905cea2ec9f47e1d82a9d01b7eeb6
I20260620 15:05:45.583117 24830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:45 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.60@KRBTEST.COM" created.
I20260620 15:05:45.607897 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.586818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.2.70.62:33151)
0620 15:05:45.587357 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:45.587363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:45.588691 (+ 1328us) server_negotiation.cc:407] Connection header received
0620 15:05:45.590877 (+ 2186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:45.590886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:45.590990 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:45.591114 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:45.592711 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.593905 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.595481 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.596242 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.596388 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:45.600536 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:45.600547 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:45.600555 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:45.600619 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:45.604436 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:45.605315 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:45.605323 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:45.605327 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:45.605478 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:45.605935 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:45.605939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:45.605941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:45.606301 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:45.606424 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:45.607321 (+ 897us) server_negotiation.cc:299] Negotiation successful
0620 15:05:45.607508 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":247,"threads_started":1}
I20260620 15:05:45.609064 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.587600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:33151 (local address 127.0.0.1:37120)
0620 15:05:45.588481 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.588547 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:45.590570 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:45.591329 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:45.591338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:45.591892 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:45.592478 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:45.592496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.594059 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.594063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:45.595325 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:45.595337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.596531 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.596536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:45.596764 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:45.597536 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:45.597590 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:45.600250 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:45.604627 (+ 4377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:45.604640 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:45.604655 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:45.605162 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:45.605623 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:45.605626 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:45.605628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:45.605825 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:45.606510 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:45.606517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:45.606901 (+ 384us) client_negotiation.cc:769] Sending connection context
0620 15:05:45.607250 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 15:05:45.607691 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":340,"threads_started":1}
I20260620 15:05:45.610984 24830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:45 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:05:45.627290 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.611837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47312)
0620 15:05:45.612073 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.612095 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:45.612216 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:45.613118 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:45.613130 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:45.613673 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:45.613905 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:45.613919 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.615343 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.615351 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:45.616578 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:45.616590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.617917 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.617922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:45.618064 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:45.618690 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:45.618707 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:45.620679 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:45.624939 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:45.624943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:45.624945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:45.625313 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:45.625755 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:45.625758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:45.625760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:45.625873 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:45.626614 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:45.626618 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:45.626741 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 15:05:45.626845 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 15:05:45.627027 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 15:05:45.627795 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.611966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.2.70.61:45601)
0620 15:05:45.612698 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:45.612705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:45.612729 (+ 24us) server_negotiation.cc:407] Connection header received
0620 15:05:45.612771 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:45.612776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:45.612957 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:45.613067 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:45.614082 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.615186 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.616813 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.617782 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.617920 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:45.620897 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:45.620914 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:45.620933 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:45.620980 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:45.624806 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:45.625442 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:45.625455 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:45.625461 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:45.625640 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:45.625977 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:45.625985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:45.625991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:45.626479 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:45.626677 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:45.627034 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 15:05:45.627318 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":423,"threads_started":1}
I20260620 15:05:45.630339 24830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:05:45.631578 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.631115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48464)
0620 15:05:45.631295 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.631403 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 15:05:45.631897 24782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
I20260620 15:05:45.633669 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.633202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48466)
0620 15:05:45.633380 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.633544 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 15:05:45.633993 24830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab.
Entry for principal kudu/127.2.70.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab.
I20260620 15:05:45.644246 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.60:35615
--webserver_interface=127.2.70.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab
--principal=kudu/127.2.70.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:38033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:33151,127.2.70.61:45601,127.2.70.60:35615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:05:45.683383 24830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 1
I20260620 15:05:45.684589 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.684125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48472)
0620 15:05:45.684339 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.684432 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260620 15:05:45.685601 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.685218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48476)
0620 15:05:45.685359 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.685447 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 15:05:45.685798 24830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
I20260620 15:05:45.748725 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } attempt: 5
I20260620 15:05:45.753994 24830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 2
I20260620 15:05:45.755031 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.754526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48490)
0620 15:05:45.754711 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.754850 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260620 15:05:45.756247 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.755834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48500)
0620 15:05:45.756002 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.756090 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 15:05:45.756554 24830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Jun 20 15:05:45 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:05:45.764182 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.749552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47316)
0620 15:05:45.749808 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.749833 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:45.749993 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:45.750379 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:45.750383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:45.750888 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:45.751063 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:45.751072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.752518 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.752522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:45.753583 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:45.753597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.754866 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.754871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:45.755060 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:45.755833 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:45.755859 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:45.758112 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:45.761791 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:45.761795 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:45.761798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:45.762189 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:45.762598 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:45.762602 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:45.762605 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:45.762738 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:45.763377 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:45.763385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:45.763542 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 15:05:45.763658 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 15:05:45.763882 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 15:05:45.764318 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.749758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.2.70.61:45601)
0620 15:05:45.749981 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:45.749987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:45.750012 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:05:45.750088 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:45.750095 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:45.750219 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:45.750347 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:45.751244 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.752346 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.753751 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:45.754679 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:45.754815 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:45.758310 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:45.758317 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:45.758322 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:45.758360 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:45.761633 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:45.762331 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:45.762337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:45.762340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:45.762482 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:45.762852 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:45.762857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:45.762860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:45.763215 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:45.763357 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:45.763799 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 15:05:45.764015 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:05:45.765352 24756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:05:45.766376 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.765968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48510)
0620 15:05:45.766179 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.766252 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 15:05:45.767222 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.766928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48526)
0620 15:05:45.767063 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.767127 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 15:05:45.767431 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
I20260620 15:05:45.818689 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 1
I20260620 15:05:45.819865 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.819398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48530)
0620 15:05:45.819601 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.819704 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:05:45.820858 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.820521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48540)
0620 15:05:45.820676 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.820757 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 15:05:45.821218 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
I20260620 15:05:45.847807 24830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 3
I20260620 15:05:45.849045 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.848495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48550)
0620 15:05:45.848736 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.848848 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:05:45.850256 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.849769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48552)
0620 15:05:45.850006 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.850100 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260620 15:05:45.850494 24830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
I20260620 15:05:45.889454 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 2
I20260620 15:05:45.890748 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.890231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48564)
0620 15:05:45.890451 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.890539 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 15:05:45.891824 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.891476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48576)
0620 15:05:45.891646 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.891715 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260620 15:05:45.892133 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
W20260620 15:05:45.980084 24838 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:05:45.980440 24838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:45.980583 24838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:45.980708 24838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:45.988047 24838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:45.988137 24838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:45.988232 24838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:45.988327 24838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:45.988430 24838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:45.993736 24830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 4
I20260620 15:05:45.994969 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.994403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48580)
0620 15:05:45.994672 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.994781 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 15:05:45.996138 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.995752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48586)
0620 15:05:45.995926 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.996008 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 15:05:45.996408 24830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
I20260620 15:05:45.998374 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 3
I20260620 15:05:45.999405 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:45.998984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48588)
0620 15:05:45.999154 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:45.999260 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:05:45.999965 24838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:38033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:33151,127.2.70.61:45601,127.2.70.60:35615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab
--principal=kudu/127.2.70.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.60:35615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.70.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:46.000447 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.000001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48600)
0620 15:05:46.000246 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:46.000348 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260620 15:05:46.000631 24756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:35615: Network error: Client connection negotiation failed: client connection to 127.2.70.60:35615: connect: Connection refused (error 111)
I20260620 15:05:46.001531 24838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:46.003038 24838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:46.015729 24846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:46.015785 24844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:46.015682 24843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:46.018906 24838 server_base.cc:1034] running on GCE node
I20260620 15:05:46.019752 24838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:46.021276 24838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:46.022532 24838 hybrid_clock.cc:648] HybridClock initialized: now 1781967946022468 us; error 39 us; skew 500 ppm
I20260620 15:05:46.022892 24838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967946, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:46.026293 24838 init.cc:377] Logged in from keytab as kudu/127.2.70.60@KRBTEST.COM (short username kudu)
I20260620 15:05:46.027896 24838 webserver.cc:460] Webserver started at http://127.2.70.60:38677/ using document root <none> and password file <none>
I20260620 15:05:46.028649 24838 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:46.028735 24838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:46.028988 24838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:46.031991 24838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/data/instance:
uuid: "613d1bb25893433a88aa12cbde4b5eb4"
format_stamp: "Formatted at 2026-06-20 15:05:46 on dist-test-slave-n5vv"
server_key: "ebca8306b67a9770a5b0c717a2434a8b"
server_key_iv: "917077eff32c537d13a5d9e3e9e959f3"
server_key_version: "encryptionkey@0"
I20260620 15:05:46.032733 24838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/wal/instance:
uuid: "613d1bb25893433a88aa12cbde4b5eb4"
format_stamp: "Formatted at 2026-06-20 15:05:46 on dist-test-slave-n5vv"
server_key: "ebca8306b67a9770a5b0c717a2434a8b"
server_key_iv: "917077eff32c537d13a5d9e3e9e959f3"
server_key_version: "encryptionkey@0"
I20260620 15:05:46.038319 24838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260620 15:05:46.042618 24853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:46.044253 24838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260620 15:05:46.044430 24838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/wal
uuid: "613d1bb25893433a88aa12cbde4b5eb4"
format_stamp: "Formatted at 2026-06-20 15:05:46 on dist-test-slave-n5vv"
server_key: "ebca8306b67a9770a5b0c717a2434a8b"
server_key_iv: "917077eff32c537d13a5d9e3e9e959f3"
server_key_version: "encryptionkey@0"
I20260620 15:05:46.044677 24838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:46.068157 24838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:46.072467 24838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:46.072682 24838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:46.092054 24838 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.60:35615
I20260620 15:05:46.092124 24904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.60:35615 every 8 connection(s)
I20260620 15:05:46.093410 24838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
I20260620 15:05:46.097142 24905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:46.099473 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 24838
I20260620 15:05:46.099650 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781967589245963-2328-0/minicluster-data/master-2/wal/instance
I20260620 15:05:46.100139 2328 external_mini_cluster.cc:1428] Setting key c1e0a92c9c50bd5a8f9aed3d886960a1
I20260620 15:05:46.105643 24905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967946, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:46.125766 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.108531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.2.70.62:33151)
0620 15:05:46.109076 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.109093 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.109915 (+ 822us) server_negotiation.cc:407] Connection header received
0620 15:05:46.110126 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.110131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.110203 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.110328 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.111856 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.112803 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.114686 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.115072 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.115162 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.119171 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.119180 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.119186 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.119242 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.122779 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.123627 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.123632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.123636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.123763 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.124137 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.124142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.124147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.124394 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.124562 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.125113 (+ 551us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.125334 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":233,"threads_started":1}
I20260620 15:05:46.126780 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.102196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.2.70.60:35615)
0620 15:05:46.103034 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.103049 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.103103 (+ 54us) server_negotiation.cc:407] Connection header received
0620 15:05:46.105437 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.105465 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.106028 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.106487 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.108036 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.110450 (+ 2414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.112297 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.113268 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.113393 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.118217 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.118230 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.118249 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.118294 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.122237 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.123127 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.123137 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.123140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.123329 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.123719 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.123727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.123733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.124511 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.125788 (+ 1277us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.125993 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.126217 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":353,"threads_started":1}
I20260620 15:05:46.126829 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.108741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:33151 (local address 127.0.0.1:37130)
0620 15:05:46.109773 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:46.109814 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:46.109976 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:46.110398 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:46.110406 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:46.111357 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:46.111691 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.111703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.112962 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.112966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.114513 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.114529 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.115315 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.115323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.115539 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.116156 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:46.116193 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:46.118940 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:46.122916 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.122927 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.122939 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.123492 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.123880 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.123885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.123889 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.124024 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.124528 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:46.124537 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:46.124767 (+ 230us) client_negotiation.cc:769] Sending connection context
0620 15:05:46.125012 (+ 245us) client_negotiation.cc:240] Negotiation successful
0620 15:05:46.125515 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"thread_start_us":490,"threads_started":1}
I20260620 15:05:46.128402 24905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967946, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:46.143337 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.129187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:45601 (local address 127.0.0.1:47318)
0620 15:05:46.129361 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:46.129380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:46.129477 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:46.129999 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:46.130007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:46.130492 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:46.130695 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.130708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.131768 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.131772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.132894 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.132908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.133679 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.133685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.133881 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.134512 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:46.134535 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:46.136855 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:46.140289 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.140297 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.140302 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.140952 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.141518 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.141524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.141528 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.141745 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.142334 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:46.142340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:46.142546 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 15:05:46.142740 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 15:05:46.143058 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:05:46.143540 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.129171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.2.70.61:45601)
0620 15:05:46.129457 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.129462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.129521 (+ 59us) server_negotiation.cc:407] Connection header received
0620 15:05:46.129616 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.129622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.129706 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.129811 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.130853 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.131619 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.133040 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.133529 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.133641 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.137053 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.137061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.137067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.137112 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.140118 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.141118 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.141194 (+ 76us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.141197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.141357 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.141866 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.141869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.141871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.142191 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.142317 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.142820 (+ 503us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.143082 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260620 15:05:46.144513 24905 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:05:46.146186 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.129641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.2.70.62:33151)
0620 15:05:46.129805 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.129810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.129916 (+ 106us) server_negotiation.cc:407] Connection header received
0620 15:05:46.130077 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.130080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.130139 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.130214 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.131505 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.132291 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.133803 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.134170 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.134335 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.139072 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.139077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.139080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.139165 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.142769 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.143865 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.143870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.143873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.144033 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.144511 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.144517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.144521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.144889 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.145220 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.145599 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.145873 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967946, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:05:46.160768 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.145617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48626 (local address 127.2.70.60:35615)
0620 15:05:46.145777 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.145782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.145796 (+ 14us) server_negotiation.cc:407] Connection header received
0620 15:05:46.145839 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.145842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.145888 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.146010 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.147176 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.147982 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.149525 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.150258 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.150373 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.154582 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.154588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.154591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.154621 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.157854 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.158713 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.158719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.158723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.158843 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.159245 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.159250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.159254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.159559 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.159728 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.160084 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.160347 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":62}
I20260620 15:05:46.161015 24910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.145371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48626)
0620 15:05:46.145549 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:46.145567 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:46.145663 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:46.146171 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:46.146174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:46.146905 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:46.147035 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.147043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.148116 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.148121 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.149339 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.149357 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.150604 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.150610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.150860 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.151625 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:46.151657 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:46.154354 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:46.157987 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.157994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.157999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.158562 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.158967 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.158972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.158977 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.159112 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.159681 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:46.159686 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:46.159836 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 15:05:46.159998 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 15:05:46.160511 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":171}
I20260620 15:05:46.163051 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.147914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.2.70.61:45601)
0620 15:05:46.148121 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.148182 (+ 61us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.148203 (+ 21us) server_negotiation.cc:407] Connection header received
0620 15:05:46.148257 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.148263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.148339 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.148445 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.149967 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.150667 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.152517 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.153114 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.153250 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.157685 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.157691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.157694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.157725 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.160300 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.161088 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.161095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.161098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.161314 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.161813 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.161816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.161818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.162056 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.162330 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.162615 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.162792 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 15:05:46.167204 24905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: Bootstrap starting.
I20260620 15:05:46.168143 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 15:05:46.171610 24905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:46.172751 24905 log.cc:824] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:46.171934 24731 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "394419b406004407a7ea5d39285fded4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37132
I20260620 15:05:46.175856 24905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: No bootstrap required, opened a new log
I20260620 15:05:46.177793 24756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 4
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967944, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
I20260620 15:05:46.186746 24905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } }
I20260620 15:05:46.187191 24905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613d1bb25893433a88aa12cbde4b5eb4, State: Initialized, Role: FOLLOWER
I20260620 15:05:46.187778 24905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } }
I20260620 15:05:46.188853 24912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.189239 24912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:46.190110 24905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:46.193624 24758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.178419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48640)
0620 15:05:46.178653 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:46.178675 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:46.178809 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:46.179225 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:46.179237 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:46.179657 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:46.179838 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.179847 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.180977 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.180983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.182389 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.182404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.183260 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.183266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.183490 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.184112 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:46.184129 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:46.186407 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:46.190599 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.190612 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.190617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.190984 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.191406 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.191412 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.191418 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.191636 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.192492 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:46.192498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:46.192675 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 15:05:46.192851 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 15:05:46.193197 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 15:05:46.193634 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.178493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.2.70.60:35615)
0620 15:05:46.178678 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.178683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.178734 (+ 51us) server_negotiation.cc:407] Connection header received
0620 15:05:46.178968 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.178977 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.179042 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.179199 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.179985 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.180829 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.182539 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.183018 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.183251 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.186585 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.186590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.186594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.186647 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.190471 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.191118 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.191125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.191130 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.191285 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.191734 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.191741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.191746 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.192227 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.192428 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.192984 (+ 556us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.193248 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 15:05:46.201325 24756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4: Bootstrap starting.
I20260620 15:05:46.205518 24756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:46.206892 24756 log.cc:824] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4: Log is configured to *not* fsync() on all Append() calls
W20260620 15:05:46.207355 24923 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:05:46.207576 24923 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:05:46.210835 24756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4: No bootstrap required, opened a new log
I20260620 15:05:46.204607 24838 master_runner.cc:186] Error getting master registration for 127.2.70.62:33151: OK, instance_id { permanent_uuid: "394419b406004407a7ea5d39285fded4" instance_seqno: 1781967945031781 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:05:46.216588 24838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:05:46.218155 24756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } }
I20260620 15:05:46.218505 24756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394419b406004407a7ea5d39285fded4, State: Initialized, Role: FOLLOWER
I20260620 15:05:46.219077 24756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } }
I20260620 15:05:46.220083 24926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.220539 24926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:46.221364 24756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:46.221410 24731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:05:46.221681 24731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:46.225957 24731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } }
I20260620 15:05:46.227872 24731 leader_election.cc:288] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [CANDIDATE]: Term 1 election: Requested vote from peers 64150268c73f413a887b53d509f7d432 (127.2.70.61:45601), 613d1bb25893433a88aa12cbde4b5eb4 (127.2.70.60:35615)
I20260620 15:05:46.229513 24880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "394419b406004407a7ea5d39285fded4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "613d1bb25893433a88aa12cbde4b5eb4"
I20260620 15:05:46.230050 24880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:46.233757 24880 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 394419b406004407a7ea5d39285fded4 in term 1.
I20260620 15:05:46.234493 24708 leader_election.cc:302] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 394419b406004407a7ea5d39285fded4, 613d1bb25893433a88aa12cbde4b5eb4; no voters:
I20260620 15:05:46.235122 24926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:46.241861 24926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [term 1 LEADER]: Becoming Leader. State: Replica: 394419b406004407a7ea5d39285fded4, State: Running, Role: LEADER
I20260620 15:05:46.243137 24926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } }
W20260620 15:05:46.251477 24937 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:05:46.251627 24937 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:05:46.258579 24940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394419b406004407a7ea5d39285fded4. Latest consensus state: current_term: 1 leader_uuid: "394419b406004407a7ea5d39285fded4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.259089 24940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: This master's current role is: LEADER
W20260620 15:05:46.270452 24710 tablet.cc:2369] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:46.278457 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.254245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47338 (local address 127.2.70.61:45601)
0620 15:05:46.254436 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.254445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.255171 (+ 726us) server_negotiation.cc:407] Connection header received
0620 15:05:46.255349 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.255354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.255437 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.255539 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.258613 (+ 3074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.259772 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.262526 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.263485 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.263618 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.268971 (+ 5353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.268981 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.268987 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.269029 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.272856 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.274256 (+ 1400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.274262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.274266 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.274562 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.275621 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.275650 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.275654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.276228 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.276576 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.277525 (+ 949us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.277912 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 15:05:46.267997 24689 master_runner.cc:186] Error getting master registration for 127.2.70.61:45601: OK, instance_id { permanent_uuid: "64150268c73f413a887b53d509f7d432" instance_seqno: 1781967945562556 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:05:46.281311 24948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 15:05:46.284629 24948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:46.290982 24689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:05:46.295687 24830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } attempt: 5
I20260620 15:05:46.298969 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.255630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.2.70.60:35615)
0620 15:05:46.255860 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.255869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.278506 (+ 22637us) server_negotiation.cc:407] Connection header received
0620 15:05:46.279882 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.279888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.279977 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.280056 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.282414 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.283526 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.285409 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.286076 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.286183 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.289284 (+ 3101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.289293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.289300 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.289346 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.295344 (+ 5998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.296402 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.296411 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.296418 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.296619 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.297145 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.297152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.297158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.297579 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.297807 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.298377 (+ 570us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.298611 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260620 15:05:46.303056 24880 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [term 1 FOLLOWER]: Refusing update from remote peer 394419b406004407a7ea5d39285fded4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:46.304953 24940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 20 15:05:46 dist-test-slave-n5vv krb5kdc[24676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967945, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
I20260620 15:05:46.309028 24909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.252155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37146 (local address 127.2.70.62:33151)
0620 15:05:46.252339 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.252348 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.253450 (+ 1102us) server_negotiation.cc:407] Connection header received
0620 15:05:46.253925 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.253932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.254015 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.254091 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.258211 (+ 4120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.259486 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.282423 (+ 22937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.283181 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.283322 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.291432 (+ 8110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.291439 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.291444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.291491 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.298071 (+ 6580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.300027 (+ 1956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.300036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.300040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.300315 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.305788 (+ 5473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.305796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.305800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.306512 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.306935 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.307984 (+ 1049us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.308396 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 15:05:46.315513 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.296296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:35615 (local address 127.0.0.1:48668)
0620 15:05:46.296725 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:46.296749 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:46.296861 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:46.299559 (+ 2698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:46.299566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:46.300079 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:46.300306 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.300320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.301784 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.301789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.302826 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:46.302840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.303921 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.303927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:46.304122 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.304857 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:46.304881 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:46.307655 (+ 2774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:46.312088 (+ 4433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.312097 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.312102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.312685 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.313272 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:46.313281 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:46.313287 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:46.313525 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:46.314344 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:46.314352 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:46.314572 (+ 220us) client_negotiation.cc:769] Sending connection context
0620 15:05:46.314746 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 15:05:46.315079 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278}
I20260620 15:05:46.316872 24908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.297130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.2.70.60:35615)
0620 15:05:46.299269 (+ 2139us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.299274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.299292 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:05:46.299341 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.299345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.299409 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.299504 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.300480 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.301631 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.303001 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.303748 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.304449 (+ 701us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.307902 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:46.307913 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:46.307921 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:46.307962 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:46.311919 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.312914 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.312922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.312928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.313124 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:46.313704 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:46.313711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:46.313728 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:46.314178 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:46.315800 (+ 1622us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.315951 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.316261 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2025,"threads_started":1}
W20260620 15:05:46.324648 24859 tablet.cc:2369] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:46.331001 24830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432: Bootstrap starting.
I20260620 15:05:46.339815 24830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:46.340921 24912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394419b406004407a7ea5d39285fded4. Latest consensus state: current_term: 1 leader_uuid: "394419b406004407a7ea5d39285fded4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.341297 24912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:46.341351 24830 log.cc:824] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:46.345075 24830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432: No bootstrap required, opened a new log
I20260620 15:05:46.354174 24912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "394419b406004407a7ea5d39285fded4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.354465 24912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:46.356921 24830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } }
I20260620 15:05:46.357331 24830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64150268c73f413a887b53d509f7d432, State: Initialized, Role: FOLLOWER
I20260620 15:05:46.357918 24830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } }
I20260620 15:05:46.360296 24926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "394419b406004407a7ea5d39285fded4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.360646 24926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:46.362891 24948 catalog_manager.cc:1347] Generated new cluster ID: b8b682ab7faf428c989aec3d369e3c65
I20260620 15:05:46.362994 24948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:46.370159 24956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.370601 24956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:46.375393 24830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:46.375550 24805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "394419b406004407a7ea5d39285fded4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "64150268c73f413a887b53d509f7d432"
I20260620 15:05:46.376065 24805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:46.380686 24805 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 394419b406004407a7ea5d39285fded4 in term 1.
I20260620 15:05:46.381590 24804 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [term 1 FOLLOWER]: Refusing update from remote peer 394419b406004407a7ea5d39285fded4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:46.383375 24940 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:05:46.395565 24948 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:46.397508 24948 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:46.418031 24948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4: Generated new TSK 0
I20260620 15:05:46.423127 24948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 15:05:46.454564 24968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "394419b406004407a7ea5d39285fded4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.455060 24968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:46.455368 24956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394419b406004407a7ea5d39285fded4. Latest consensus state: current_term: 1 leader_uuid: "394419b406004407a7ea5d39285fded4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.455618 24956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:46.455742 24965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "394419b406004407a7ea5d39285fded4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "394419b406004407a7ea5d39285fded4" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 33151 } } peers { permanent_uuid: "64150268c73f413a887b53d509f7d432" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 45601 } } peers { permanent_uuid: "613d1bb25893433a88aa12cbde4b5eb4" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 35615 } } }
I20260620 15:05:46.456185 24965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394419b406004407a7ea5d39285fded4 [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:46.474850 24974 catalog_manager.cc:1259] Loaded cluster ID: b8b682ab7faf428c989aec3d369e3c65
I20260620 15:05:46.475049 24974 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432: loading cluster ID for follower catalog manager: success
I20260620 15:05:46.479462 24974 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432: acquiring CA information for follower catalog manager: success
I20260620 15:05:46.482055 24974 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64150268c73f413a887b53d509f7d432: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 15:05:46.595178 24835 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:46.586388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.2.70.61:45601)
0620 15:05:46.586633 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.586640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.593162 (+ 6522us) server_negotiation.cc:407] Connection header received
0620 15:05:46.593460 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.593467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.593614 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.593691 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.594911 (+ 1220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47346: BlockingRecv error: recv got EOF from 127.0.0.1:47346 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 15:05:46.597777 24909 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:46.585842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.2.70.62:33151)
0620 15:05:46.586087 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.586095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.596191 (+ 10096us) server_negotiation.cc:407] Connection header received
0620 15:05:46.596459 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.596465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.596553 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.596622 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.597553 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37158: BlockingRecv error: recv got EOF from 127.0.0.1:37158 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260620 15:05:46.611575 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:46.586860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48676 (local address 127.2.70.60:35615)
0620 15:05:46.587126 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.587134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.599630 (+ 12496us) server_negotiation.cc:407] Connection header received
0620 15:05:46.609798 (+ 10168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.609805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.609904 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.609992 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.611189 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48676: BlockingRecv error: recv got EOF from 127.0.0.1:48676 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 15:05:46.666702 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.646437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.2.70.61:45601)
0620 15:05:46.646620 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.646627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.658048 (+ 11421us) server_negotiation.cc:407] Connection header received
0620 15:05:46.658359 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.658365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.658608 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.658766 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:46.659531 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.660788 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.663557 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.664239 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.664350 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.665111 (+ 761us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:46.665869 (+ 758us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:46.665995 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.666244 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.666349 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:05:46.735482 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:46.719168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.2.70.60:35615)
0620 15:05:46.719362 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.719369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.733626 (+ 14257us) server_negotiation.cc:407] Connection header received
0620 15:05:46.733955 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.733961 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.734079 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.734150 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.735242 (+ 1092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48684: BlockingRecv error: recv got EOF from 127.0.0.1:48684 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 15:05:46.789912 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.772784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.2.70.61:45601)
0620 15:05:46.772986 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.772992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.782818 (+ 9826us) server_negotiation.cc:407] Connection header received
0620 15:05:46.783337 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.783351 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.783465 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.783538 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:46.784256 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.785392 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.787470 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.788166 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.788283 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.788981 (+ 698us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:46.789149 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:46.789226 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.789465 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.789583 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:05:46.831498 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:46.822263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.2.70.60:35615)
0620 15:05:46.822485 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.822492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.829402 (+ 6910us) server_negotiation.cc:407] Connection header received
0620 15:05:46.829944 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.829952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.830062 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.830147 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.831243 (+ 1096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48700: BlockingRecv error: recv got EOF from 127.0.0.1:48700 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 15:05:46.886227 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.868095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47372 (local address 127.2.70.61:45601)
0620 15:05:46.868314 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.868323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.878991 (+ 10668us) server_negotiation.cc:407] Connection header received
0620 15:05:46.879570 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.879576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.879673 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.879749 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:46.880513 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.881736 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.883787 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.884550 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.884656 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.885310 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:46.885462 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:46.885546 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.885788 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.885878 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260620 15:05:46.923463 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:46.916354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.2.70.60:35615)
0620 15:05:46.916561 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.916570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.921702 (+ 5132us) server_negotiation.cc:407] Connection header received
0620 15:05:46.921957 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.921964 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.922059 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.922164 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.923230 (+ 1066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48706: BlockingRecv error: recv got EOF from 127.0.0.1:48706 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 15:05:46.938115 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.931161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.2.70.61:45601)
0620 15:05:46.931397 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.931403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.931807 (+ 404us) server_negotiation.cc:407] Connection header received
0620 15:05:46.932007 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.932013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.932116 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.932240 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:46.932706 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.933905 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.935842 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.936591 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.936724 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.937249 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:46.937392 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:46.937461 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.937737 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.937826 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260620 15:05:46.949844 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:46.947020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.2.70.60:35615)
0620 15:05:46.947294 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.947300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.947802 (+ 502us) server_negotiation.cc:407] Connection header received
0620 15:05:46.948106 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.948112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.948213 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.948304 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.949585 (+ 1281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48708: BlockingRecv error: recv got EOF from 127.0.0.1:48708 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
I20260620 15:05:46.969002 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.962435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.2.70.61:45601)
0620 15:05:46.962658 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.962664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.963028 (+ 364us) server_negotiation.cc:407] Connection header received
0620 15:05:46.963281 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.963290 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.963378 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.963487 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:46.963929 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.964775 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.966363 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:46.967068 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:46.967213 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:46.967775 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:46.967935 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:46.968024 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:46.968457 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 15:05:46.968552 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260620 15:05:46.977846 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:46.975135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48714 (local address 127.2.70.60:35615)
0620 15:05:46.975349 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.975354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:46.976182 (+ 828us) server_negotiation.cc:407] Connection header received
0620 15:05:46.976465 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:46.976470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:46.976546 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:46.976675 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:46.977593 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48714: BlockingRecv error: recv got EOF from 127.0.0.1:48714 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 15:05:47.005400 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:46.999450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47388 (local address 127.2.70.61:45601)
0620 15:05:46.999702 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:46.999711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.000290 (+ 579us) server_negotiation.cc:407] Connection header received
0620 15:05:47.000531 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.000535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.000608 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.000714 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:47.001200 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.002142 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.003496 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.003928 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.004072 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:47.004481 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:47.004709 (+ 228us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:47.004827 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:47.004977 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 15:05:47.005075 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260620 15:05:47.014348 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:47.011932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.2.70.60:35615)
0620 15:05:47.012176 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.012185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.012940 (+ 755us) server_negotiation.cc:407] Connection header received
0620 15:05:47.013161 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.013168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.013260 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.013374 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:47.014159 (+ 785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48722: BlockingRecv error: recv got EOF from 127.0.0.1:48722 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:05:47.057875 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:47.051055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47396 (local address 127.2.70.61:45601)
0620 15:05:47.051254 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.051261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.051817 (+ 556us) server_negotiation.cc:407] Connection header received
0620 15:05:47.052043 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.052050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.052161 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.052261 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:47.052746 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.053880 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.056036 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.056513 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.056617 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:47.057094 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:47.057206 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:47.057269 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:47.057424 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 15:05:47.057540 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 15:05:47.065235 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:47.063245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48732 (local address 127.2.70.60:35615)
0620 15:05:47.063495 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.063504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.063883 (+ 379us) server_negotiation.cc:407] Connection header received
0620 15:05:47.064038 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.064042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.064117 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.064206 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:47.065013 (+ 807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48732: BlockingRecv error: recv got EOF from 127.0.0.1:48732 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 15:05:47.138988 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:47.132389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.2.70.61:45601)
0620 15:05:47.132563 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.132568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.132947 (+ 379us) server_negotiation.cc:407] Connection header received
0620 15:05:47.133104 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.133141 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.133216 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.133304 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:47.133865 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.134636 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.136744 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.137512 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.137678 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:47.138258 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:47.138415 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:47.138520 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:47.138697 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 15:05:47.138779 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 15:05:47.147687 24950 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:47.145763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48740 (local address 127.2.70.60:35615)
0620 15:05:47.145990 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.145998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.146331 (+ 333us) server_negotiation.cc:407] Connection header received
0620 15:05:47.146540 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.146545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.146648 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.146757 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:47.147486 (+ 729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48740: BlockingRecv error: recv got EOF from 127.0.0.1:48740 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:05:47.209860 24923 catalog_manager.cc:1259] Loaded cluster ID: b8b682ab7faf428c989aec3d369e3c65
I20260620 15:05:47.209996 24923 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: loading cluster ID for follower catalog manager: success
I20260620 15:05:47.212568 24923 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: acquiring CA information for follower catalog manager: success
I20260620 15:05:47.214376 24923 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 613d1bb25893433a88aa12cbde4b5eb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:05:47.284333 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:47.279336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.2.70.61:45601)
0620 15:05:47.279530 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.279535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.279808 (+ 273us) server_negotiation.cc:407] Connection header received
0620 15:05:47.279969 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.279973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.280052 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.280163 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:47.280607 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.281408 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.282789 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.283241 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.283343 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:47.283709 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:47.283815 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:47.283889 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:47.284014 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 15:05:47.284093 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 15:05:47.297016 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:47.290707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48746 (local address 127.2.70.60:35615)
0620 15:05:47.290957 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.290964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.291315 (+ 351us) server_negotiation.cc:407] Connection header received
0620 15:05:47.291465 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.291469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.291724 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.291821 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:47.292243 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.293135 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.294719 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.295163 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.295285 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:47.295780 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:47.296378 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:47.296497 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:47.296606 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 15:05:47.296680 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 15:05:47.321018 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:47.303694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37166 (local address 127.2.70.62:33151)
0620 15:05:47.314415 (+ 10721us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.314421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.314446 (+ 25us) server_negotiation.cc:407] Connection header received
0620 15:05:47.314576 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.314584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.314837 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.315049 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:47.315885 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.316847 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.318871 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.319299 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.319440 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:47.319971 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:47.320326 (+ 355us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:47.320457 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:47.320597 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 15:05:47.320683 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10585,"thread_start_us":10731,"threads_started":1}
I20260620 15:05:47.322858 24950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:47.304963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.2.70.60:35615)
0620 15:05:47.305328 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.305337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.315168 (+ 9831us) server_negotiation.cc:407] Connection header received
0620 15:05:47.316031 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.316039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.316125 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.316221 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:47.317042 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.318207 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.320655 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.321366 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.321470 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:47.322056 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:47.322211 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:47.322294 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:47.322441 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 15:05:47.322519 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 15:05:47.323274 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:47.304472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.2.70.61:45601)
0620 15:05:47.304696 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:47.304703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:47.315761 (+ 11058us) server_negotiation.cc:407] Connection header received
0620 15:05:47.316018 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:47.316023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:47.316098 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:47.316153 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:47.317194 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.318068 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.320666 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:47.321444 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:47.321573 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:47.322305 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:47.322448 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:47.322512 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:47.322863 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 15:05:47.322951 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 15:05:47.326798 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24689
I20260620 15:05:47.328451 24751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:47.441550 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24689
W20260620 15:05:47.462049 24781 connection.cc:535] server connection from 127.0.0.1:47316 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 15:05:47.462419 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24763
I20260620 15:05:47.463478 24825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:47.571233 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24763
I20260620 15:05:47.591725 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 24838
I20260620 15:05:47.592917 24900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:47.700040 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 24838
2026-06-20T15:05:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4501 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 15:05:47 dist-test-slave-n5vv krb5kdc[25087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 15:05:47 dist-test-slave-n5vv krb5kdc[25087](info): set up 2 sockets
Jun 20 15:05:47 dist-test-slave-n5vv krb5kdc[25087](info): commencing operation
krb5kdc: starting...
W20260620 15:05:48.898097 2328 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/test-user.keytab.
Jun 20 15:05:49 dist-test-slave-n5vv krb5kdc[25087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T15:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T15:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.70.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
Entry for principal kudu/127.2.70.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab.
I20260620 15:05:49.163619 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34717
--webserver_interface=127.2.70.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34717,127.2.70.61:34251,127.2.70.60:37915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 15:05:49.501384 25100 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:05:49.501781 25100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:49.501852 25100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:49.501935 25100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 15:05:49.509286 25100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:49.509378 25100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:49.509435 25100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:49.509528 25100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:49.509616 25100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:49.519604 25100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34717,127.2.70.61:34251,127.2.70.60:37915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.62.keytab
--principal=kudu/127.2.70.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.62:34717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.70.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:49.520877 25100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:49.522238 25100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:49.534363 25108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:49.534500 25105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:49.534484 25106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:49.535415 25100 server_base.cc:1034] running on GCE node
I20260620 15:05:49.536197 25100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:49.537773 25100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:49.539047 25100 hybrid_clock.cc:648] HybridClock initialized: now 1781967949538976 us; error 48 us; skew 500 ppm
I20260620 15:05:49.539507 25100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:49 dist-test-slave-n5vv krb5kdc[25087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967949, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:49.547354 25100 init.cc:377] Logged in from keytab as kudu/127.2.70.62@KRBTEST.COM (short username kudu)
I20260620 15:05:49.549129 25100 webserver.cc:460] Webserver started at http://127.2.70.62:33153/ using document root <none> and password file <none>
I20260620 15:05:49.549932 25100 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:49.550064 25100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:49.550392 25100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:49.553465 25100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/data/instance:
uuid: "bc689247d85440cca14450c737ee230e"
format_stamp: "Formatted at 2026-06-20 15:05:49 on dist-test-slave-n5vv"
server_key: "b9e3c03546dbde287ef53ca29da64307"
server_key_iv: "f2a18095fe6fb9570ea8c215f100468b"
server_key_version: "encryptionkey@0"
I20260620 15:05:49.554355 25100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/wal/instance:
uuid: "bc689247d85440cca14450c737ee230e"
format_stamp: "Formatted at 2026-06-20 15:05:49 on dist-test-slave-n5vv"
server_key: "b9e3c03546dbde287ef53ca29da64307"
server_key_iv: "f2a18095fe6fb9570ea8c215f100468b"
server_key_version: "encryptionkey@0"
I20260620 15:05:49.559612 25100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 15:05:49.563294 25115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:49.564744 25100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 15:05:49.564949 25100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/wal
uuid: "bc689247d85440cca14450c737ee230e"
format_stamp: "Formatted at 2026-06-20 15:05:49 on dist-test-slave-n5vv"
server_key: "b9e3c03546dbde287ef53ca29da64307"
server_key_iv: "f2a18095fe6fb9570ea8c215f100468b"
server_key_version: "encryptionkey@0"
I20260620 15:05:49.565192 25100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:49.581645 25100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:49.586201 25100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:49.586493 25100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:49.605844 25100 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.62:34717
I20260620 15:05:49.605862 25166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.62:34717 every 8 connection(s)
I20260620 15:05:49.607246 25100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/data/info.pb
I20260620 15:05:49.609874 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 25100
I20260620 15:05:49.610065 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-0/wal/instance
I20260620 15:05:49.610601 2328 external_mini_cluster.cc:1428] Setting key 93c9ea1f6cf1f40254df1688b78c692d
I20260620 15:05:49.611680 25167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:49.622002 25167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:49 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967949, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.70.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.61@KRBTEST.COM" created.
I20260620 15:05:49.648494 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.625323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.2.70.62:34717)
0620 15:05:49.626266 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:49.626286 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:49.626711 (+ 425us) server_negotiation.cc:407] Connection header received
0620 15:05:49.628757 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:49.628762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:49.629014 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:49.629237 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:49.630597 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:49.631805 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:49.634390 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:49.635065 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:49.635287 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:49.639990 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:49.640004 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:49.640029 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:49.640074 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:49.643491 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:49.644280 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:49.644287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:49.644289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:49.644444 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:49.644898 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:49.644905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:49.644914 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:49.645474 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:49.645765 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:49.646353 (+ 588us) server_negotiation.cc:299] Negotiation successful
0620 15:05:49.646819 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":679,"thread_start_us":466,"threads_started":1}
I20260620 15:05:49.648515 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.625323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34717 (local address 127.0.0.1:53990)
0620 15:05:49.626540 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.626586 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:49.628361 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:49.629257 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:49.629264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:49.630029 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:49.630373 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:49.630391 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:49.632023 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:49.632032 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:49.634167 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:49.634189 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:49.635420 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:49.635431 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:49.635689 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:49.636519 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:49.636553 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:49.639584 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:49.643655 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:49.643671 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:49.643686 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:49.644118 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:49.644590 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:49.644595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:49.644599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:49.644773 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:49.645688 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:49.645704 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:49.645939 (+ 235us) client_negotiation.cc:769] Sending connection context
0620 15:05:49.646208 (+ 269us) client_negotiation.cc:240] Negotiation successful
0620 15:05:49.646957 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":214,"spinlock_wait_cycles":17664,"thread_start_us":804,"threads_started":1}
I20260620 15:05:49.651902 25167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:05:49.653264 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.652784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54230)
0620 15:05:49.652969 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.653087 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 15:05:49.653595 25119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
I20260620 15:05:49.655221 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.654854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54240)
0620 15:05:49.655018 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.655089 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:05:49.655431 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:34251: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab.
Entry for principal kudu/127.2.70.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab.
I20260620 15:05:49.675290 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.61:34251
--webserver_interface=127.2.70.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab
--principal=kudu/127.2.70.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34717,127.2.70.61:34251,127.2.70.60:37915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:05:49.704798 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } attempt: 1
I20260620 15:05:49.706009 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.705562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54248)
0620 15:05:49.705763 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.705877 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260620 15:05:49.707327 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.706880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54256)
0620 15:05:49.707056 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.707163 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 15:05:49.707559 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:34251: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
I20260620 15:05:49.775818 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } attempt: 2
I20260620 15:05:49.777554 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.776723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54272)
0620 15:05:49.777053 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.777295 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260620 15:05:49.778709 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.778268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54280)
0620 15:05:49.778431 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.778537 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 15:05:49.778945 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:34251: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
I20260620 15:05:49.870262 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } attempt: 3
I20260620 15:05:49.871628 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.870990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54284)
0620 15:05:49.871278 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.871419 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 15:05:49.872957 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:49.872434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54296)
0620 15:05:49.872626 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:49.872747 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 15:05:49.873250 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:34251: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
I20260620 15:05:50.016541 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } attempt: 4
W20260620 15:05:50.016947 25174 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:05:50.017431 25174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:50.017566 25174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:50.017705 25174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:05:50.018033 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.017341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54312)
0620 15:05:50.017776 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.017892 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260620 15:05:50.019186 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.018785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54318)
0620 15:05:50.018960 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.019058 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 15:05:50.019414 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.61:34251: Network error: Client connection negotiation failed: client connection to 127.2.70.61:34251: connect: Connection refused (error 111)
W20260620 15:05:50.026731 25174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:50.026844 25174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:50.026994 25174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:50.027087 25174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:50.027196 25174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:50.038355 25174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34717,127.2.70.61:34251,127.2.70.60:37915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.61.keytab
--principal=kudu/127.2.70.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.61:34251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.70.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:50.039762 25174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:50.041263 25174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 15:05:50.053893 25180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:50.053916 25179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:50.053848 25182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:50.054746 25174 server_base.cc:1034] running on GCE node
I20260620 15:05:50.055583 25174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:50.057091 25174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:50.058343 25174 hybrid_clock.cc:648] HybridClock initialized: now 1781967950058268 us; error 44 us; skew 500 ppm
I20260620 15:05:50.058691 25174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967950, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:50.062052 25174 init.cc:377] Logged in from keytab as kudu/127.2.70.61@KRBTEST.COM (short username kudu)
I20260620 15:05:50.063619 25174 webserver.cc:460] Webserver started at http://127.2.70.61:37201/ using document root <none> and password file <none>
I20260620 15:05:50.064335 25174 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:50.064468 25174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:50.064798 25174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:50.067866 25174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/data/instance:
uuid: "fab8823ffa18421a8de75159c794fe74"
format_stamp: "Formatted at 2026-06-20 15:05:50 on dist-test-slave-n5vv"
server_key: "56369fa4127819067d8d42e7d08af29e"
server_key_iv: "72f3e20d711621adc2889ca573968d2f"
server_key_version: "encryptionkey@0"
I20260620 15:05:50.068694 25174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/wal/instance:
uuid: "fab8823ffa18421a8de75159c794fe74"
format_stamp: "Formatted at 2026-06-20 15:05:50 on dist-test-slave-n5vv"
server_key: "56369fa4127819067d8d42e7d08af29e"
server_key_iv: "72f3e20d711621adc2889ca573968d2f"
server_key_version: "encryptionkey@0"
I20260620 15:05:50.074250 25174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260620 15:05:50.077654 25189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:50.079012 25174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 15:05:50.079223 25174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/wal
uuid: "fab8823ffa18421a8de75159c794fe74"
format_stamp: "Formatted at 2026-06-20 15:05:50 on dist-test-slave-n5vv"
server_key: "56369fa4127819067d8d42e7d08af29e"
server_key_iv: "72f3e20d711621adc2889ca573968d2f"
server_key_version: "encryptionkey@0"
I20260620 15:05:50.079454 25174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 15:05:50.097186 25174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:50.101827 25174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:50.102119 25174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:50.121449 25174 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.61:34251
I20260620 15:05:50.121470 25240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.61:34251 every 8 connection(s)
I20260620 15:05:50.122884 25174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/data/info.pb
I20260620 15:05:50.126947 25241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:50.132521 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 25174
I20260620 15:05:50.132714 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-1/wal/instance
I20260620 15:05:50.133239 2328 external_mini_cluster.cc:1428] Setting key 7c1cb58e3852332c57a768cdfaa0d8b4
I20260620 15:05:50.134299 25241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967950, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:50.158800 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.137073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.2.70.62:34717)
0620 15:05:50.137334 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.137339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.138763 (+ 1424us) server_negotiation.cc:407] Connection header received
0620 15:05:50.140955 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.140963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.141078 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.141195 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.142934 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.144113 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.145905 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.146652 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.146795 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.150787 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.150798 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.150806 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.150857 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.155213 (+ 4356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.156077 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.156096 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.156099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.156225 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.156717 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.156722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.156724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.157020 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.157267 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.158139 (+ 872us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.158388 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260620 15:05:50.159893 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.137679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34717 (local address 127.0.0.1:53998)
0620 15:05:50.138546 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.138597 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:50.140607 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:50.141421 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:50.141430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:50.142123 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:50.142739 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.142759 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.144297 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.144302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.145739 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.145753 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.146929 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.146935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.147231 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.148030 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:50.148053 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:50.150489 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:50.155390 (+ 4901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.155398 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.155411 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.155919 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.156356 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.156363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.156369 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.156559 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.157232 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:50.157240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:50.157709 (+ 469us) client_negotiation.cc:769] Sending connection context
0620 15:05:50.158002 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 15:05:50.158606 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":407,"threads_started":1}
WARNING: no policy specified for kudu/127.2.70.60@KRBTEST.COM; defaulting to no policy
I20260620 15:05:50.161936 25241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.70.60@KRBTEST.COM" created.
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967950, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:05:50.181097 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.162836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54324)
0620 15:05:50.163040 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.163057 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:50.163194 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:50.164359 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:50.164364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:50.164886 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:50.165073 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.165085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.167426 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.167435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.169128 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.169148 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.170106 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.170115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.170394 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.171039 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:50.171070 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:50.173831 (+ 2761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:50.178647 (+ 4816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.178652 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.178655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.179009 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.179466 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.179469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.179471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.179600 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.180390 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:50.180393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:50.180509 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 15:05:50.180618 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 15:05:50.180825 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 15:05:50.181747 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.163138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54324 (local address 127.2.70.61:34251)
0620 15:05:50.163882 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.163889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.163916 (+ 27us) server_negotiation.cc:407] Connection header received
0620 15:05:50.163967 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.163974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.164158 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.164347 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.165335 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.167241 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.169279 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.169953 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.170054 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.174079 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.174098 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.174118 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.174175 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.178506 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.179157 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.179170 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.179176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.179328 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.179716 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.179724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.179731 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.180250 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.180467 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.180837 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.181175 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":421,"threads_started":1}
I20260620 15:05:50.184605 25241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:05:50.185977 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.185436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32904)
0620 15:05:50.185680 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.185813 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260620 15:05:50.186302 25192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
I20260620 15:05:50.188014 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.187662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32914)
0620 15:05:50.187829 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.187911 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 15:05:50.188266 25241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.70.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab.
Entry for principal kudu/127.2.70.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab.
I20260620 15:05:50.204669 2328 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
/tmp/dist-test-taskK9gJB7/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.60:37915
--webserver_interface=127.2.70.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab
--principal=kudu/127.2.70.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.70.20:37325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.70.62:34717,127.2.70.61:34251,127.2.70.60:37915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 15:05:50.237643 25241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 1
I20260620 15:05:50.238907 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.238430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32918)
0620 15:05:50.238644 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.238742 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260620 15:05:50.240025 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.239605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32928)
0620 15:05:50.239823 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.239906 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260620 15:05:50.240268 25241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
I20260620 15:05:50.308525 25241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 2
I20260620 15:05:50.309862 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.309255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32932)
0620 15:05:50.309476 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.309692 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:05:50.311034 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.310667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32934)
0620 15:05:50.310836 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.310908 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 15:05:50.311317 25241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
I20260620 15:05:50.318677 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } attempt: 5
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967949, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:05:50.332669 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.319334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54336)
0620 15:05:50.319558 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.319581 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:50.319745 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:50.320125 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:50.320131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:50.320646 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:50.320872 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.320885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.321979 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.321985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.323333 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.323348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.324171 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.324178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.324355 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.324981 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:50.325003 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:50.327319 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:50.330326 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.330333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.330337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.330730 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.331082 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.331087 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.331091 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.331228 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.331730 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:50.331735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:50.331887 (+ 152us) client_negotiation.cc:769] Sending connection context
0620 15:05:50.332030 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:05:50.332304 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 15:05:50.332669 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.319423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.2.70.61:34251)
0620 15:05:50.319584 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.319590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.319667 (+ 77us) server_negotiation.cc:407] Connection header received
0620 15:05:50.319872 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.319876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.319957 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.320039 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.321017 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.321826 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.323501 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.323989 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.324102 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.327508 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.327513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.327516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.327546 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.330188 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.330851 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.330858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.330866 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.330974 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.331332 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.331339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.331342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.331617 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.331765 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.332135 (+ 370us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.332337 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:05:50.333665 25167 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:05:50.334689 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.334252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32950)
0620 15:05:50.334431 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.334524 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:05:50.335629 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.335273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32958)
0620 15:05:50.335412 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.335509 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 15:05:50.335875 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
I20260620 15:05:50.387136 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 1
I20260620 15:05:50.388453 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.387872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32964)
0620 15:05:50.388118 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.388239 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 15:05:50.389778 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.389308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32970)
0620 15:05:50.389534 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.389633 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 15:05:50.390043 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
I20260620 15:05:50.402570 25241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 3
I20260620 15:05:50.403823 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.403318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32974)
0620 15:05:50.403564 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.403662 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 15:05:50.405133 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.404635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32984)
0620 15:05:50.404874 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.404954 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260620 15:05:50.405368 25241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
I20260620 15:05:50.458356 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 2
I20260620 15:05:50.459739 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.459076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32986)
0620 15:05:50.459370 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.459500 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260620 15:05:50.460996 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.460526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32992)
0620 15:05:50.460719 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.460827 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 15:05:50.461309 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
W20260620 15:05:50.545900 25248 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 15:05:50.546274 25248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 15:05:50.546370 25248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 15:05:50.546523 25248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 15:05:50.548604 25241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 4
I20260620 15:05:50.549932 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.549321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:32994)
0620 15:05:50.549671 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.549770 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 15:05:50.551034 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.550657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:33008)
0620 15:05:50.550824 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.550912 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 15:05:50.551399 25241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
W20260620 15:05:50.554863 25248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 15:05:50.554986 25248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 15:05:50.555116 25248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 15:05:50.555207 25248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 15:05:50.555299 25248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 15:05:50.565310 25248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.70.20:37325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.70.62:34717,127.2.70.61:34251,127.2.70.60:37915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/krb5kdc/kudu_127.2.70.60.keytab
--principal=kudu/127.2.70.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.70.60:37915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.70.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 20 Jun 2026 13:41:20 UTC on ccycloud.quasar-vdagoi.root.comops.site
build id 4457462
ASAN enabled
I20260620 15:05:50.566771 25248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 15:05:50.567559 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 3
I20260620 15:05:50.568204 25248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 15:05:50.568713 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.568224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:33014)
0620 15:05:50.568466 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.568551 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 15:05:50.569878 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.569449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:33016)
0620 15:05:50.569703 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.569778 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260620 15:05:50.570093 25167 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.70.60:37915: Network error: Client connection negotiation failed: client connection to 127.2.70.60:37915: connect: Connection refused (error 111)
W20260620 15:05:50.580829 25253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:50.581254 25256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 15:05:50.581282 25254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 15:05:50.582240 25248 server_base.cc:1034] running on GCE node
I20260620 15:05:50.583156 25248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 15:05:50.584805 25248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 15:05:50.586076 25248 hybrid_clock.cc:648] HybridClock initialized: now 1781967950585973 us; error 80 us; skew 500 ppm
I20260620 15:05:50.586506 25248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967950, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 15:05:50.589975 25248 init.cc:377] Logged in from keytab as kudu/127.2.70.60@KRBTEST.COM (short username kudu)
I20260620 15:05:50.591655 25248 webserver.cc:460] Webserver started at http://127.2.70.60:33765/ using document root <none> and password file <none>
I20260620 15:05:50.592485 25248 fs_manager.cc:359] Metadata directory not provided
I20260620 15:05:50.592638 25248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 15:05:50.592998 25248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 15:05:50.596328 25248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/data/instance:
uuid: "ffa28dc5002b4895a6d6ea4fa19e7156"
format_stamp: "Formatted at 2026-06-20 15:05:50 on dist-test-slave-n5vv"
server_key: "1fb5a1818d039899b99cb86c2c99fce5"
server_key_iv: "d5202c0e37cc38f412be7f1c13f555e1"
server_key_version: "encryptionkey@0"
I20260620 15:05:50.597186 25248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/wal/instance:
uuid: "ffa28dc5002b4895a6d6ea4fa19e7156"
format_stamp: "Formatted at 2026-06-20 15:05:50 on dist-test-slave-n5vv"
server_key: "1fb5a1818d039899b99cb86c2c99fce5"
server_key_iv: "d5202c0e37cc38f412be7f1c13f555e1"
server_key_version: "encryptionkey@0"
I20260620 15:05:50.603029 25248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 15:05:50.607131 25263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 15:05:50.608691 25248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 15:05:50.608904 25248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/data,/tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/wal
uuid: "ffa28dc5002b4895a6d6ea4fa19e7156"
format_stamp: "Formatted at 2026-06-20 15:05:50 on dist-test-slave-n5vv"
server_key: "1fb5a1818d039899b99cb86c2c99fce5"
server_key_iv: "d5202c0e37cc38f412be7f1c13f555e1"
server_key_version: "encryptionkey@0"
I20260620 15:05:50.609146 25248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 15:05:50.610816 25163 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.2.70.62:34717
I20260620 15:05:50.657189 25248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 15:05:50.661684 25248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 15:05:50.661967 25248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 15:05:50.680505 25248 rpc_server.cc:307] RPC server started. Bound to: 127.2.70.60:37915
I20260620 15:05:50.680524 25314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.70.60:37915 every 8 connection(s)
I20260620 15:05:50.681982 25248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/data/info.pb
I20260620 15:05:50.686021 25315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 15:05:50.691897 2328 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu as pid 25248
I20260620 15:05:50.692088 2328 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskK9gJB7/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781967589245963-2328-0/minicluster-data/master-2/wal/instance
I20260620 15:05:50.692632 2328 external_mini_cluster.cc:1428] Setting key 359f8baba729b2b393b6924606b3d6cf
I20260620 15:05:50.692819 25315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967950, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
I20260620 15:05:50.713064 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.695663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.2.70.62:34717)
0620 15:05:50.696267 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.696273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.696781 (+ 508us) server_negotiation.cc:407] Connection header received
0620 15:05:50.697790 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.697794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.697856 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.697946 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.699352 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.700208 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.701938 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.702484 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.702575 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.706730 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.706735 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.706738 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.706772 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.709923 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.710637 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.710646 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.710649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.710772 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.711272 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.711276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.711279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.711565 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.711728 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.712392 (+ 664us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.712613 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":304,"threads_started":1}
I20260620 15:05:50.714134 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.694471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33022 (local address 127.2.70.60:37915)
0620 15:05:50.695218 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.695228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.695264 (+ 36us) server_negotiation.cc:407] Connection header received
0620 15:05:50.697526 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.697552 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.697855 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.698057 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.699498 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.700848 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.702284 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.702948 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.703059 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.707159 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.707169 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.707185 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.707230 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.710219 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.710893 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.710903 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.710906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.711065 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.711467 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.711471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.711474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.711991 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.712331 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.712651 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.712894 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":554,"thread_start_us":358,"threads_started":1}
I20260620 15:05:50.714164 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.695674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.62:34717 (local address 127.0.0.1:54010)
0620 15:05:50.696614 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.696664 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:50.697526 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:50.698034 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:50.698040 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:50.698816 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:50.699154 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.699171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.700363 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.700369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.701805 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.701816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.702730 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.702738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.703058 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.704087 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:50.704116 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:50.706530 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:50.710060 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.710073 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.710084 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.710502 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.710892 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.710896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.710901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.711116 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.711721 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:50.711730 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:50.712022 (+ 292us) client_negotiation.cc:769] Sending connection context
0620 15:05:50.712268 (+ 246us) client_negotiation.cc:240] Negotiation successful
0620 15:05:50.712798 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":155,"thread_start_us":517,"threads_started":1}
I20260620 15:05:50.715947 25315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967950, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.62@KRBTEST.COM
I20260620 15:05:50.730393 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.716728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.61:34251 (local address 127.0.0.1:54350)
0620 15:05:50.716894 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.716913 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:50.717005 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:50.717868 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:50.717874 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:50.718423 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:50.718627 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.718640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.719619 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.719628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.720491 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.720502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.721275 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.721281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.721518 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.722179 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:50.722201 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:50.724491 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:50.727970 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.727977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.727981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.728350 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.728727 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.728732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.728735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.728876 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.729394 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:50.729398 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:50.729595 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 15:05:50.729738 (+ 143us) client_negotiation.cc:240] Negotiation successful
0620 15:05:50.730018 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 15:05:50.730420 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.717203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.2.70.61:34251)
0620 15:05:50.717613 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.717617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.717632 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:05:50.717679 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.717683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.717739 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.717818 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.718787 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.719455 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.720653 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.721123 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.721260 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.724694 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.724703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.724711 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.724753 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.727816 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.728468 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.728475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.728478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.728615 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.728990 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.728995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.728998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.729267 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.729427 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.729828 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.730039 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260620 15:05:50.731580 25315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } has no permanent_uuid. Determining permanent_uuid...
I20260620 15:05:50.732372 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.717386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.2.70.62:34717)
0620 15:05:50.717583 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.717589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.717607 (+ 18us) server_negotiation.cc:407] Connection header received
0620 15:05:50.717739 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.717744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.717850 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.717938 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.719372 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.720031 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.721259 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.721762 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.721902 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.726676 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.726683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.726690 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.726736 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.729340 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.730016 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.730029 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.730031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.730142 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.730710 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.730715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.730718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.731015 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.731330 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.731752 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.731974 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967950, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.60@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967949, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.70.61@KRBTEST.COM
I20260620 15:05:50.745893 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.732493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.2.70.60:37915)
0620 15:05:50.732657 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.732662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.732779 (+ 117us) server_negotiation.cc:407] Connection header received
0620 15:05:50.732918 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.732923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.732982 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.733077 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.733852 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.735000 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.736288 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.736695 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.736792 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.740315 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.740324 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.740329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.740372 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.743372 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.743991 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.743996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.744006 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.744121 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.744487 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.744492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.744496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.744767 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.744916 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.745260 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.745481 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 15:05:50.746011 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.732434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:33034)
0620 15:05:50.732696 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.732712 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:50.732817 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:50.733092 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:50.733095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:50.733558 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:50.733733 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.733746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.735141 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.735145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.736149 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.736164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.736844 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.736850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.737028 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.737661 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:50.737678 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:50.740103 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:50.743504 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.743512 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.743516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.743868 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.744234 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.744241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.744246 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.744378 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.744879 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:50.744884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:50.745024 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 15:05:50.745174 (+ 150us) client_negotiation.cc:240] Negotiation successful
0620 15:05:50.745668 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":152}
I20260620 15:05:50.747289 25167 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 4
I20260620 15:05:50.749334 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.734164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54364 (local address 127.2.70.61:34251)
0620 15:05:50.734339 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.734343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.734358 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:05:50.734396 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.734399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.734488 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.734610 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.735880 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.736736 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.737999 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.738647 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.738796 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.743546 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.743551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.743554 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.743590 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.746498 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.747171 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.747177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.747180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.747309 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.747741 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.747749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.747755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.748131 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.748412 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.748732 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.748973 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 15:05:50.752295 25315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: Bootstrap starting.
I20260620 15:05:50.754185 2328 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967949, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.62@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
I20260620 15:05:50.756507 25315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:50.758025 25315 log.cc:824] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:50.757254 25142 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "bc689247d85440cca14450c737ee230e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54012
I20260620 15:05:50.762027 25315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: No bootstrap required, opened a new log
I20260620 15:05:50.764014 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.747943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:33040)
0620 15:05:50.748189 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.748228 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:50.748336 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:50.748915 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:50.748922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:50.749498 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:50.749723 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.749736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.751297 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.751302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.752737 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.752754 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.753782 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.753788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.754000 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.754710 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:50.754738 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:50.756954 (+ 2216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:50.761001 (+ 4047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.761005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.761027 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.761627 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.762035 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.762040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.762043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.762231 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.762956 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:50.762961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:50.763143 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 15:05:50.763290 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 15:05:50.763616 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260620 15:05:50.764036 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.747963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.2.70.60:37915)
0620 15:05:50.748568 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.748578 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.748593 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:05:50.748643 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.748648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.748732 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.749876 (+ 1144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.749994 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.751111 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.752942 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.753612 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.753747 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.757125 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.757131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.757135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.757171 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.760772 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.761754 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.761760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.761763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.761915 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.762347 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.762352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.762355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.762827 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.763008 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.763429 (+ 421us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.763654 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474}
I20260620 15:05:50.770231 25167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e: Bootstrap starting.
I20260620 15:05:50.772895 25315 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } }
I20260620 15:05:50.773347 25315 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa28dc5002b4895a6d6ea4fa19e7156, State: Initialized, Role: FOLLOWER
I20260620 15:05:50.773936 25315 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } }
I20260620 15:05:50.774389 25167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:50.775274 25322 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:50.775673 25322 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:50.775746 25167 log.cc:824] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:50.776425 25315 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:50.779227 25167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e: No bootstrap required, opened a new log
W20260620 15:05:50.786557 25334 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 15:05:50.786692 25334 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 15:05:50.787040 25167 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } }
I20260620 15:05:50.787441 25167 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc689247d85440cca14450c737ee230e, State: Initialized, Role: FOLLOWER
I20260620 15:05:50.787954 25167 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } }
I20260620 15:05:50.788892 25336 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:50.789352 25336 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:50.790184 25167 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:50.784320 25248 master_runner.cc:186] Error getting master registration for 127.2.70.62:34717: OK, instance_id { permanent_uuid: "bc689247d85440cca14450c737ee230e" instance_seqno: 1781967949600290 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:05:50.793591 25248 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:05:50.793702 25142 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 15:05:50.793865 25142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:50.796047 25142 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } }
I20260620 15:05:50.797576 25142 leader_election.cc:288] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [CANDIDATE]: Term 1 election: Requested vote from peers fab8823ffa18421a8de75159c794fe74 (127.2.70.61:34251), ffa28dc5002b4895a6d6ea4fa19e7156 (127.2.70.60:37915)
I20260620 15:05:50.798064 25290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc689247d85440cca14450c737ee230e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156"
I20260620 15:05:50.798540 25290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:50.802124 25290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc689247d85440cca14450c737ee230e in term 1.
I20260620 15:05:50.802932 25118 leader_election.cc:302] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc689247d85440cca14450c737ee230e, ffa28dc5002b4895a6d6ea4fa19e7156; no voters:
I20260620 15:05:50.803560 25336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 15:05:50.817116 25336 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [term 1 LEADER]: Becoming Leader. State: Replica: bc689247d85440cca14450c737ee230e, State: Running, Role: LEADER
I20260620 15:05:50.818761 25336 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } }
I20260620 15:05:50.835843 25343 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc689247d85440cca14450c737ee230e. Latest consensus state: current_term: 1 leader_uuid: "bc689247d85440cca14450c737ee230e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:50.836596 25343 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:50.839331 25354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260620 15:05:50.839404 25121 tablet.cc:2369] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:50.851132 25241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } attempt: 5
I20260620 15:05:50.852329 25354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 15:05:50.876184 25290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [term 1 FOLLOWER]: Refusing update from remote peer bc689247d85440cca14450c737ee230e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:50.877269 25343 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:05:50.880903 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.833771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.2.70.61:34251)
0620 15:05:50.833982 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.833989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.843200 (+ 9211us) server_negotiation.cc:407] Connection header received
0620 15:05:50.843618 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.843625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.843726 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.843801 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.849066 (+ 5265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.850260 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.857608 (+ 7348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.858316 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.858432 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.868254 (+ 9822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.868262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.868267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.868330 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.872030 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.877662 (+ 5632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.877667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.877671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.877891 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.878346 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.878354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.878358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.878799 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.879050 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.880329 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.880552 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Jun 20 15:05:50 dist-test-slave-n5vv krb5kdc[25087](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781967950, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.70.61@KRBTEST.COM for kudu/127.2.70.60@KRBTEST.COM
I20260620 15:05:50.892308 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.821722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.2.70.60:37915)
0620 15:05:50.821915 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.821922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.827030 (+ 5108us) server_negotiation.cc:407] Connection header received
0620 15:05:50.828646 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.828655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.828786 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.828880 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.839375 (+ 10495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.840419 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.847000 (+ 6581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.847760 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.847875 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.872971 (+ 25096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.872979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.872984 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.873026 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.876720 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.879518 (+ 2798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.879523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.879526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.879710 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.882178 (+ 2468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.882182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.882185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.882531 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.882785 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.891467 (+ 8682us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.891790 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 15:05:50.900658 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.825424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.2.70.62:34717)
0620 15:05:50.825685 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.825695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.835448 (+ 9753us) server_negotiation.cc:407] Connection header received
0620 15:05:50.836019 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.836025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.836120 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.836206 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.843186 (+ 6980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.844297 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.850090 (+ 5793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.850823 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.850925 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.890333 (+ 39408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.890344 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.890351 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.890425 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.895118 (+ 4693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.896276 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.896283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.896294 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.896487 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.897055 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.897063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.897069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.897628 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.898127 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.899729 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.900053 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260620 15:05:50.914042 25269 tablet.cc:2369] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 15:05:50.915889 25243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.851712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.70.60:37915 (local address 127.0.0.1:33064)
0620 15:05:50.852193 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 15:05:50.852220 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 15:05:50.852349 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 15:05:50.872478 (+ 20129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 15:05:50.872488 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 15:05:50.873180 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0620 15:05:50.873467 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.873503 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.875823 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.875829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.877416 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 15:05:50.877438 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.881276 (+ 3838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.881286 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 15:05:50.881610 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.883840 (+ 2230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 15:05:50.883873 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0620 15:05:50.887652 (+ 3779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 15:05:50.911780 (+ 24128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.911786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.911791 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.912467 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.912958 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 15:05:50.912965 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 15:05:50.912970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 15:05:50.913131 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 15:05:50.914847 (+ 1716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 15:05:50.914854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 15:05:50.915047 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 15:05:50.915200 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 15:05:50.915502 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301}
I20260620 15:05:50.917363 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:50.852596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33064 (local address 127.2.70.60:37915)
0620 15:05:50.871856 (+ 19260us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:50.871866 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:50.871898 (+ 32us) server_negotiation.cc:407] Connection header received
0620 15:05:50.871980 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:50.871989 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:50.872098 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:50.873679 (+ 1581us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:50.874035 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.875644 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.877681 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:50.878537 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:50.887919 (+ 9382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:50.888250 (+ 331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 15:05:50.888259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 15:05:50.888267 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 15:05:50.888309 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 15:05:50.911580 (+ 23271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.912650 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.912656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.912659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.912829 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 15:05:50.913290 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 15:05:50.913295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 15:05:50.913299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 15:05:50.913670 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 15:05:50.916563 (+ 2893us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:50.916691 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 15:05:50.916919 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19050,"thread_start_us":19628,"threads_started":1}
I20260620 15:05:50.906395 25100 master_runner.cc:186] Error getting master registration for 127.2.70.61:34251: OK, instance_id { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" instance_seqno: 1781967950115893 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 15:05:50.918125 25100 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 15:05:50.923854 25241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74: Bootstrap starting.
I20260620 15:05:50.927963 25241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74: Neither blocks nor log segments found. Creating new log.
I20260620 15:05:50.929299 25241 log.cc:824] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74: Log is configured to *not* fsync() on all Append() calls
I20260620 15:05:50.933792 25322 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc689247d85440cca14450c737ee230e. Latest consensus state: current_term: 1 leader_uuid: "bc689247d85440cca14450c737ee230e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:50.934130 25322 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:50.939013 25241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74: No bootstrap required, opened a new log
I20260620 15:05:50.942621 25359 mvcc.cc:205] Tried to move back new op lower bound from 7298940726771949568 to 7298940726568169472. Current Snapshot: MvccSnapshot[applied={T|T < 7298940726771949568}]
I20260620 15:05:50.951217 25241 raft_consensus.cc:348] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } }
I20260620 15:05:50.951778 25241 raft_consensus.cc:729] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab8823ffa18421a8de75159c794fe74, State: Initialized, Role: FOLLOWER
I20260620 15:05:50.952342 25241 consensus_queue.cc:260] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } }
I20260620 15:05:50.959447 25366 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:50.961912 25366 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:50.961786 25241 sys_catalog.cc:564] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 15:05:50.962217 25215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 15:05:50.966367 25215 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [term 1 FOLLOWER]: Refusing update from remote peer bc689247d85440cca14450c737ee230e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 15:05:50.966588 25216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bc689247d85440cca14450c737ee230e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fab8823ffa18421a8de75159c794fe74"
I20260620 15:05:50.967826 25336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 15:05:50.975696 25336 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc689247d85440cca14450c737ee230e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:50.976016 25336 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:50.976406 25322 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc689247d85440cca14450c737ee230e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:50.976734 25322 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:50.978883 25216 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc689247d85440cca14450c737ee230e in term 1.
I20260620 15:05:50.981936 25354 catalog_manager.cc:1347] Generated new cluster ID: 831ccb3dd98f4cf8b1b6abb9392aa8b2
I20260620 15:05:50.982036 25354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 15:05:51.045591 25366 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc689247d85440cca14450c737ee230e. Latest consensus state: current_term: 1 leader_uuid: "bc689247d85440cca14450c737ee230e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:51.045964 25366 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:51.046772 25376 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bc689247d85440cca14450c737ee230e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:51.051944 25376 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 15:05:51.054793 25381 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 15:05:51.076223 25343 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bc689247d85440cca14450c737ee230e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc689247d85440cca14450c737ee230e" member_type: VOTER last_known_addr { host: "127.2.70.62" port: 34717 } } peers { permanent_uuid: "fab8823ffa18421a8de75159c794fe74" member_type: VOTER last_known_addr { host: "127.2.70.61" port: 34251 } } peers { permanent_uuid: "ffa28dc5002b4895a6d6ea4fa19e7156" member_type: VOTER last_known_addr { host: "127.2.70.60" port: 37915 } } }
I20260620 15:05:51.076578 25343 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e [sys.catalog]: This master's current role is: LEADER
I20260620 15:05:51.080704 25354 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 15:05:51.085841 25354 catalog_manager.cc:1504] Loading token signing keys...
I20260620 15:05:51.103590 25354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc689247d85440cca14450c737ee230e: Generated new TSK 0
I20260620 15:05:51.106142 25354 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 15:05:51.187042 25358 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.180847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33066 (local address 127.2.70.60:37915)
0620 15:05:51.181329 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.181338 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.184994 (+ 3656us) server_negotiation.cc:407] Connection header received
0620 15:05:51.185241 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.185251 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.185417 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.185588 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.186785 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33066: BlockingRecv error: recv got EOF from 127.0.0.1:33066 (error 108)
Metrics: {"server-negotiator.queue_time_us":323}
W20260620 15:05:51.189008 25319 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.180581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.2.70.62:34717)
0620 15:05:51.181137 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.181146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.186174 (+ 5028us) server_negotiation.cc:407] Connection header received
0620 15:05:51.187496 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.187501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.187582 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.187660 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.188769 (+ 1109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54018: BlockingRecv error: recv got EOF from 127.0.0.1:54018 (error 108)
Metrics: {"server-negotiator.queue_time_us":347}
W20260620 15:05:51.189256 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.186698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.2.70.61:34251)
0620 15:05:51.186912 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.186917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.187746 (+ 829us) server_negotiation.cc:407] Connection header received
0620 15:05:51.187967 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.187974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.188073 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.188154 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.189028 (+ 874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54388: BlockingRecv error: recv got EOF from 127.0.0.1:54388 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260620 15:05:51.198805 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.196555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.2.70.61:34251)
0620 15:05:51.196736 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.196744 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.197227 (+ 483us) server_negotiation.cc:407] Connection header received
0620 15:05:51.197474 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.197519 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.197640 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.197765 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.198558 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54390: BlockingRecv error: recv got EOF from 127.0.0.1:54390 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 15:05:51.207398 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.203266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.2.70.61:34251)
0620 15:05:51.203441 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.203447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.206053 (+ 2606us) server_negotiation.cc:407] Connection header received
0620 15:05:51.206271 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.206277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.206362 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.206469 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.207183 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54400: BlockingRecv error: recv got EOF from 127.0.0.1:54400 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 15:05:51.214241 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.212103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.2.70.61:34251)
0620 15:05:51.212305 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.212314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.212802 (+ 488us) server_negotiation.cc:407] Connection header received
0620 15:05:51.213013 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.213018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.213098 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.213200 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.213999 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54414: BlockingRecv error: recv got EOF from 127.0.0.1:54414 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260620 15:05:51.225092 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.222934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.2.70.61:34251)
0620 15:05:51.223193 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.223202 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.223731 (+ 529us) server_negotiation.cc:407] Connection header received
0620 15:05:51.223946 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.223951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.224046 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.224157 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.224859 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54416: BlockingRecv error: recv got EOF from 127.0.0.1:54416 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260620 15:05:51.238562 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.236368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.2.70.61:34251)
0620 15:05:51.236622 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.236631 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.237142 (+ 511us) server_negotiation.cc:407] Connection header received
0620 15:05:51.237375 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.237381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.237466 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.237609 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.238331 (+ 722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54428: BlockingRecv error: recv got EOF from 127.0.0.1:54428 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260620 15:05:51.259848 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.257665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.2.70.61:34251)
0620 15:05:51.257886 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.257896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.258419 (+ 523us) server_negotiation.cc:407] Connection header received
0620 15:05:51.258651 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.258655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.258731 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.258849 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.259603 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54434: BlockingRecv error: recv got EOF from 127.0.0.1:54434 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 15:05:51.297247 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.294973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.2.70.61:34251)
0620 15:05:51.295238 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.295248 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.295801 (+ 553us) server_negotiation.cc:407] Connection header received
0620 15:05:51.296037 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.296042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.296124 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.296232 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.296989 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54444: BlockingRecv error: recv got EOF from 127.0.0.1:54444 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260620 15:05:51.384158 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.380097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54454 (local address 127.2.70.61:34251)
0620 15:05:51.380315 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.380321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.382296 (+ 1975us) server_negotiation.cc:407] Connection header received
0620 15:05:51.382516 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.382522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.382630 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.382754 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.383871 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54454: BlockingRecv error: recv got EOF from 127.0.0.1:54454 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260620 15:05:51.518744 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.516732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.2.70.61:34251)
0620 15:05:51.517003 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.517011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.517319 (+ 308us) server_negotiation.cc:407] Connection header received
0620 15:05:51.517586 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.517591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.517665 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.517778 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.518524 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50298: BlockingRecv error: recv got EOF from 127.0.0.1:50298 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260620 15:05:51.780596 25245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 15:05:51.778769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.2.70.61:34251)
0620 15:05:51.779009 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:51.779015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:51.779214 (+ 199us) server_negotiation.cc:407] Connection header received
0620 15:05:51.779388 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:51.779392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:51.779464 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:51.779553 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 15:05:51.780391 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50308: BlockingRecv error: recv got EOF from 127.0.0.1:50308 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 15:05:51.788602 25334 catalog_manager.cc:1259] Loaded cluster ID: 831ccb3dd98f4cf8b1b6abb9392aa8b2
I20260620 15:05:51.788694 25334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: loading cluster ID for follower catalog manager: success
I20260620 15:05:51.791366 25334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: acquiring CA information for follower catalog manager: success
I20260620 15:05:51.793119 25334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ffa28dc5002b4895a6d6ea4fa19e7156: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:05:52.059151 25381 catalog_manager.cc:1259] Loaded cluster ID: 831ccb3dd98f4cf8b1b6abb9392aa8b2
I20260620 15:05:52.059276 25381 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74: loading cluster ID for follower catalog manager: success
I20260620 15:05:52.061975 25381 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74: acquiring CA information for follower catalog manager: success
I20260620 15:05:52.064112 25381 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fab8823ffa18421a8de75159c794fe74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 15:05:52.302994 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:52.296580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.2.70.61:34251)
0620 15:05:52.297559 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:52.297564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:52.297579 (+ 15us) server_negotiation.cc:407] Connection header received
0620 15:05:52.297642 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:52.297645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:52.297792 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:52.297929 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:52.298426 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.299269 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.300578 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.301039 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.301202 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:52.301723 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:52.302313 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:52.302520 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:52.302578 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 15:05:52.302663 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":576,"threads_started":1}
I20260620 15:05:52.319859 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:52.313823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.2.70.60:37915)
0620 15:05:52.314551 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:52.314559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:52.314579 (+ 20us) server_negotiation.cc:407] Connection header received
0620 15:05:52.314719 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:52.314723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:52.314876 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:52.315058 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:52.315467 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.316253 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.317640 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.318044 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.318216 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:52.318720 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:52.319245 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:52.319414 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:52.319471 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 15:05:52.319554 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":489,"threads_started":1}
I20260620 15:05:52.348161 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:52.326252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.2.70.62:34717)
0620 15:05:52.339013 (+ 12761us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:52.339023 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:52.339698 (+ 675us) server_negotiation.cc:407] Connection header received
0620 15:05:52.339836 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:52.339844 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:52.340092 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:52.340284 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:52.341272 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.342416 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.344738 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.345544 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.345688 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:52.346425 (+ 737us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:52.346913 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:52.347021 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:52.347726 (+ 705us) server_negotiation.cc:299] Negotiation successful
0620 15:05:52.347818 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12577,"thread_start_us":12776,"threads_started":1}
I20260620 15:05:52.348584 25454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:52.326884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.2.70.60:37915)
0620 15:05:52.327055 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:52.327061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:52.339742 (+ 12681us) server_negotiation.cc:407] Connection header received
0620 15:05:52.340027 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:52.340033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:52.340135 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:52.340202 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:52.341313 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.342688 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.345726 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.346558 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.346691 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:52.347546 (+ 855us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:52.347736 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:52.347811 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:52.348067 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 15:05:52.348174 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 15:05:52.356529 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 15:05:52.339916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.2.70.61:34251)
0620 15:05:52.340100 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 15:05:52.340105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 15:05:52.347624 (+ 7519us) server_negotiation.cc:407] Connection header received
0620 15:05:52.349049 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 15:05:52.349055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 15:05:52.349148 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 15:05:52.349223 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 15:05:52.350489 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.351650 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.353652 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 15:05:52.354340 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 15:05:52.354474 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 15:05:52.355331 (+ 857us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 15:05:52.355482 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 15:05:52.355590 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 15:05:52.356106 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 15:05:52.356190 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 15:05:52.357538 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 25100
I20260620 15:05:52.358583 25162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:52.471706 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 25100
I20260620 15:05:52.493016 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 25174
I20260620 15:05:52.494194 25236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:52.604702 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 25174
I20260620 15:05:52.625394 2328 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu pid 25248
I20260620 15:05:52.626572 25310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260620 15:05:52.734531 2328 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskK9gJB7/build/asan/bin/kudu with pid 25248
2026-06-20T15:05:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5025 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9526 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (363451 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST